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and error with the practical application of SSIS to solve real problems. We 
learned by doing and then rethinking to look for better solutions. Things 
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learned since then. 

Data Integration Life Cycle Management with SSIS grew out of 
the many iterations working on various projects. It’s not just about 
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CHAPTER 1 


Introduction to DIML 


DevOps is a combination of the words “Development” and “Operations.” 
DevOps is about process improvement, which manifests in faster time to 
market, higher quality, repeatable automation, and code that is easier to 
support and maintain. Software testing is a major part of DevOps, starting 
with unit-testing conducted by software developers. Software testing 
occurs at all enterprise application tiers (Development, Quality Assurance, 
User Acceptance, Production, etc.). DevOps developers support 
Operations by surfacing process instrumentation and building repeatable 
configurations scripts. Configurations scripting supports higher quality 
and faster disaster recovery, and rapidly and reliably adding enterprise 
application tiers. 

Software developers follow best practices to build robust enterprise 
software. For decades, developers have applied a collection of best 
practices called application lifecycle management, or ALM, when building 
applications. What is ALM? Application lifecycle management includes 
design principles that support DevOps. An important development 
concept is separation of concerns, design best practices including 
externalization (parameterization) and decoupling. 

Data integration is moving data from one location to another. Data 
is often collected from disparate sources and loaded into a database to 
support centralized reporting. The reporting databases are known by 
different names in different enterprises: operational data store (ODS), 
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staging database, central repository, data warehouse (DW), or enterprise 
data warehouse (EDW), to list a few. 

Data integration is software development. Data integration lifecycle 
management, or DILM, is the art and science of managing data integration 
in the modern DevOps enterprise. In this book, I share what I’ve learned 
managing SQL Server Integration Services (SSIS) data integration solutions 
in enterprises large and small. 

My goal in writing this book is to teach you how to manage SSIS in your 
enterprise. As with all other software development platforms, lifecycle 
management is a best practice with SSIS. As with all other software 
development, it’s possible to practice DevOps with SSIS. 


Some History 


SQL Server Integration Services (SSIS) was released in November 2005. 
Development on SSIS started long before, and I’m told the product was 
originally slated to be dubbed Data Transformation Services (DTS) 2.0. 
Many assemblies sport the namespace “Dts.” 

I want to introduce this book by making a few statements about SSIS. 


SSIS Is a Software Development Platform 


I cannot recall how many times I’ve seen job postings for database 
administrators “with SSIS experience.’ 


Database administrator, database developer, and SSIS developer are 
different roles. 
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I cringe a little when I read said postings. SSIS is a software 
development platform and I am a software developer. I am also a database 
developer. I am not a database administrator (DBA). I’ve tried to do the job 
of a DBA and failed. Miserably. It’s not that I don’t appreciate the role of 
a DBA; I promise I do. Some of my best friends are DBAs. I want to begin 
by expressing to you this simple truth: DBA, database developer, and SSIS 
developer are different roles. 


SSIS Is an Enterprise Data Integration Engine 


SSIS is designed to move data from one location to another, which is 

the essence of data integration. Data integration can include reshaping, 
cleansing, and transformation of data; but at its heart, data integration is 
data relocation. The SSIS Data Flow Task was revolutionary when it was 
introduced. In my humble opinion, it remains pretty slick technology. The 
pipeline architecture surfaces most of the levers one needs to tweak to 
achieve enterprise scale. 

Is SSIS the perfect data integration engine for every data integration 
need? No. But it is an amazingly flexible solution to most data integration 
requirements. 

A number of times I’ve accomplished what-cannot-be-done with 
SSIS. Years ago I even got into a flame war online with a really smart, 
internet-famous software developer who wrote a post listing all the things 
wrong with SSIS. I shared with this individual that I teach people SSIS and 
all my students know the solution to almost every item listed in the post. 


Someone recently asked me, “When is Microsoft going to deliver 
an enterprise data integration engine?” My response? “2005.” 


- Andy Leonard 
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SSIS Is Difficult to Learn 


How do I know? I (and others) have made a living for more than a decade 
teaching people how to use SSIS. 

All software platforms have “corners,” - quirks and edge cases that 
the language just isn’t the best at managing. SSIS has about 30 corners. I 
describe the data flow task to students in this manner: “SSIS wants you to 
think like a data flow and thinking like a data flow is hard.” 


“SSIS wants you to think like a data flow.” 


Lifecycle Management 


SSIS, even at the time of this writing, is difficult to manage in the enterprise 
lifecycle. Exhibit A is comparing SSIS packages. SSIS is XML-based. XML 

is a self-describing data format with less respect for order than traditional 
data stores. For example, the following XML snippets are equivalent: 


<Book> 
<Title>Data Integration Lifecycle Management</Title> 
<Author>Andy Leonard</Author> 
<Year>2017</Year> 

</Book> 

<Book> 
<Author>Andy Leonard</Author> 
<Title>Data Integration Lifecycle Management</Title> 
<Year>2017</Year> 

</Book> 


You may look at this example and quip, “But Andy, I can figure out 
that these data are equivalent just by looking at them.’ You are correct; 
examining five rows of data is pretty simple. Imagine looking at five 
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hundred rows of XML, with tags and attributes in a different order, and you 
begin to understand the complexity of comparing XML data. 

XML is not bad or wrong. XMLs semi-structured nature makes SSIS 
difficult to compare. 


Solutions and Credit Where Credit Is Due 


The remainder of this book focuses on solutions. With the exception of the 
BimlExpress Metadata Framework, Kent Bradshaw, Kevin Hazzard, and I 
developed the solutions contained in this book. Scott Currie and his team 
at Varigence, Inc. built Business Intelligence Markup Language (Biml) and 
taught us how to author Biml. Some of what Scott and his team taught us 
made its way into the BimlExpress Metadata Framework. 

A more accurate rendering of the facts is that Kent, Kevin, and I learned 
what we know from experience and from others. In a very real sense, none 
of us is self-taught. Rather, we are community-taught. 

Some of these solutions are simply ways of using the technology 
Microsoft shipped “in the box” with SSIS. Some are best practices. Some 
are manual and others are automated. Some are free tools and utilities 
by vendors. Some are free utilities and tools the team at Enterprise Data 
& Analytics (entdna.com), Tudor Data Solutions (tudords.com), and 
DevJourney (devjourney.com) have developed; many are part of the 
DILM Suite (dilmsuite.com). One, BimlExpress, is a third-party product 
from Varigence (varigence.com). A couple, SSIS Framework Community 
Edition and BimlExpress Metadata Framework, are free versions of for-sale 
implemented solutions that Enterprise Data & Analytics sells as part of 
consulting engagements. 

Do we have all the answers? Goodness no! We have some. Like you, 
we learn new stuff every day. Here in this book is some of what we know 
today. 


CHAPTER 2 


SSIS 


This book is not intended to teach you SSIS. If you read this book and work 
through the examples, you may learn more about SSIS. That is my hope, 
but it’s not my goal in writing this book to teach you SSIS. My goal is teach 
you how to manage SSIS in your enterprise. If you desire to learn SSIS, 
I recommend the Stairway to Integration Services at SQL Server Central 
(sqlservercentral.com/stairway/72494/) for beginners and SQL Server 
Integration Services Design Patterns (amazon. com/Server-Integration- 
Services -Design-Patterns/dp/1484200837) for more advanced learning. 

Learning by example is best. In this chapter, you will build an SSIS 
project for demonstration purposes. This SSIS project will include one 
SSIS package, a connection manager, variables, project parameters, 
and package parameters. You will use this SSIS project, DILMSample, 
throughout the remainder of this book. The SSIS project is built in SQL 
Server Data Tools (SSDT). Please see Appendix A for links to the tools and 
utilities you will use throughout this book. 

I will discuss data integration instrumentation and messaging to 
surface log messages during the execution of the SSIS package. These 
messages serve people troubleshooting failed executions and surface 


important data integration instrumentation metadata. 
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The Demo 


Open SSDT and create a new SSIS project named DILMSample. When 
SSDT creates a new SSIS project, it loads a default SSIS project. The default 
SSIS project includes 


e Project parameters, stored in the Project .params file 
e A virtual folder for project-scope connection managers 
e A virtual folder for package parts 
e Avirtual folder for control flow package parts 
e Avirtual folder for miscellaneous items 
e A default (empty) SSIS package named Package. dtsx 


Rename Package. dtsx as SimplePackage.dtsx, as shown in Figure 2-1. 


peng Ree? E : Pu — l Ed ai o #7 
Solution Explorer “Ox 
| | i 

G\o-sap|s—= 
Search Solution Explorer (Ctri+;) p~ 


fg.) Solution 'DILMSample' (1 project) 
4 ¿E DILMSample 
&@ Project.params 
fa) Connection Managers 
4 ‘wl SSIS Packages 






Ey SimplePackage.dtsx 
4 ‘al Package Parts 
fa) Control Flow 
ta! Miscellaneous 


Figure 2-1. The DILMSample SSIS project 


Renaming the package communicates intent; leaving the default 


package name communicates laziness. 
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Adding Package Parameters 


Since we plan to use this project and package to demonstrate lifecycle 
management, let’s add parameters. To start, click the Parameters tab on 
the SimplePackage.dtsx package. Add two package parameters with the 
settings shown in Table 2-1. 


Table 2-1. Package Parameter Settings 


Name Datatype Value Sensitive Required Description 
IntPkgParam Int32 42 False False 
StringPkgParam String Hi There! False False 


When completed, your Parameters tab should appear similar to that 
shown in Figure 2-2. 


SimplePackage.dtsx [Design] > > [Is 





o ControlFlow gig DataFlow MaE acces E] EventHandlers "= Package Explorer 
ee 

Name Data type Value Sensitive Required Description 
ý IntPkgParam Int32 42 False False 

Q StringPkgParam String Hi There! . False | False 


Biro er tet tert ett 


Figure 2-2. Package parameters 


You add these demo parameters at the package scope. Scope 
is an important concept in software development and lifecycle 
management. The “Dev” part of DevOps focuses on building code that 
is easily manageable by Operations. How does scope help? Later you 
will externalize parameter values into Transact-SQL (T-SQL) scripts. 
Operations personnel will ultimately manage externalized values 
by deploying and maintaining T-SQL scripts. One goal of DevOps 


development is to communicate as much information as possible to 
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Operations people, especially Operations people who have no idea how 
SSIS works. Scoping parameters and variables to their proper level (no 
higher or lower) communicates where and how parameter values are used 
in the SSIS application, thus shedding some light on the “black box” of the 
data integration project. 


Adding Project Parameters 


Open the Project Parameters window by double-clicking the Project. 
params artifact in Solution Explorer, shown in Figure 2-3. 


Solution Explorer 


@\o-S@a\s— 


fg.) Solution 'DILMSample’ (loading...) 


| a 4& DILMSample 
| Q Project.params — 
onnection Managers 


a tm) SSIS Packages 
EL SimplePackage.dtsx 
a tm) Package Parts 
ta) Control Flow 
fa) Miscellaneous 





Figure 2-3. Project parameters 


Add two project parameters with the settings from Table 2-2. 
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Table 2-2. Project Parameter Settings 


Name Datatype Value Sensitive Required Description 


IntProjParam Int32 -99 False False 
StringProjParam String A project False False 
parameter 


When completed your Project Parameters tab should appear similar to 


that shown in Figure 2-4. 


ee eM Project.params [Design] > 


oy pP 

Name Data type Value Sensitive Required Description 

&@ = intProjParam int32 -99 False False ESA 
@ StringProjParam Sting A project parameter False False Eons 


Figure 2-4. Project parameters 


SSIS parameters work a lot like SSIS variables. There are two important 


distinctions: 


1. SSIS variable values may be changed at any time, 
before or during SSIS package execution. SSIS 
parameter values may be changed before SSIS 
package execution but not during SSIS package 


execution. 


2. SSIS parameter values may be marked as sensitive 
or required. Sensitive parameters are encrypted. 
Required parameters must be overridden before 


SSIS package execution. 
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A Note About Variables, Parameters, and Scope 


Create SSIS variables for values that will change during SSIS package 
execution, such as variables used in Foreach Loop container assignments. 
Create package-scope parameters for values used in a single SSIS 
package. Package-scope parameter values cannot change during SSIS 
package execution. Create project-scope parameters for values that are 
shared across all (or several) SSIS packages in the SSIS project. 
Save and close the Project Parameters window. 


Adding an Execute SQL Task 


Add an Execute SQL Task to the control flow of the SimplePackage.dtsx 
SSIS package. Open the Execute SQL Task Editor and set the 
ConnectionType property to ADO.NET. Click the Connection property and 
then click <New connection...>. The ADO.Net Connection Manager Editor 
opens. You can configure the ADO.Net connection manager to connect 

to any SQL Server database you desire. You are going to execute a generic 
query that will work with any SQL Server relational database. 

I am using a virtual machine named vmDemo. I have an instance of 
SQL Server 2016 installed named vmDemo\Demo. I configured my ADO.Net 
connection manager to connect to a database named TestDB, as shown in 
Figure 2-5. 
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‘S Connection Manager X 
Provider: | Net Providers\SqlClient Data Provider ~ 





Server name: 


'vmDemo\Demo {v Refresh 





Log on to the server 


Authentication: Windows Authentication 


Save my password 
Connect to a database 


@) Select or enter a database name: 
TestDB) 


© Attach a database file: 


Test Connection Cancel Help 


Figure 2-5. Configuring an ADO.Net connection manager 


Click OK until you return to the Execute SQL Task Editor. Click in the 
Value textbox of the SQLStatement property and then click the ellipsis 
to open the Enter SQL Query dialog. In this dialog enter the following 
Transact-SQL statement: 


Select Count(*) As TableCount 
From [sys].[tables | 


Why alias the value of Count (*)? The Execute SQL task expects return 
values in a tabular format with column names, especially if assigned to an 
SSIS variable (which is next). 
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Change the ResultSet property from None to Single Row. Click 
the Result Set page in the listbox on the left side of the Execute SQL 
Task Editor. Click the Add button and change the Result Name from 
NewResultName to 0. Click the dropdown in the Variable Name column 
beside the 0 Result Name and click <New variable...> to open the Add 
Variable dialog. Make sure the Container is set to SimplePackage. Name 
the variable TableCount and set the Value type property to Int32. Supply 


a default Value of 0. Your Add Variable dialog should appear as shown in 
Figure 2-6. 


g Add Variable x 


Specify the properties of the new variable. 


Container: | SimplePackage v | 


Name: 


Namespace: User 
Value type: Int32 ai 
= 
L] Read only 


Figure 2-6. Adding a new SSIS variable 


You may select the scope of the SSIS variable using the Container 
dropdown in the Add Variable dialog. Pay attention to this setting. If the 
Execute SQL task resides in a Sequence container, it is easy to accidentally 


scope a variable to the host Sequence container instead of the SSIS 
package. 
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A Note About SSIS Variable Scope 


Because SSIS variables are rarely scoped beneath the SSIS package, the 
Microsoft SSIS Development Team changed the default behavior for 
SSIS variable scope. Before the change, SSIS variable scope defaulted to 
the executable with focus. SSIS variable scope now defaults to the SSIS 
package, except in Add Variable dialogs. 

Click the OK button. Your Execute SQL Task Editor’s Result Set page 
should appear as shown in Figure 2-7. 


LS Execute SQL Task Editor 


ir _ Configure the properties required to run SQL statements and stored 
General | Result Name Variable Name 
Parameter Mapping | peme 
Result Set | Rikat AN SEa mT ira a aAa AA e a 


Figure 2-7. A Result Set, configured 


Click the OK button to close the Execute SQL Task Editor. Right-click 
the Execute SQL task and click Rename. Rename the task “SQL Get Table 
Count” 


Adding a Script Task 


Add a Script task to SimplePackage.dtsx’s Control Flow. Rename it “SCR 
Log Values” and connect an OnSuccess precedence constraint from the 
SQL Get Table Count Execute SQL task to the SCR Log Values Script task. 
Open the Script Task Editor. You can select Microsoft Visual Basic as the 
ScriptLanguage property or accept the default of Microsoft Visual C#. The 
demos in this book will use C#. 
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Why C#? 


Since late 2016 Microsoft has repeatedly communicated that Visual Basic 
and C# language functionality will diverge (blogs.msdn.microsoft.com/ 
dotnet/2017/02/01/the-net-language-strategy/). Microsoft’s stated 
opinion (from the link) is that it will continue to evolve C# as a “state of the 
art programming language” while other features won't be added to Visual 
Basic because “they wouldn’t address a need or fit naturally in VB.” Please 
read the post and the links contained therein. C# and VB will be different 
moving forward, and C# will get features that will not be available in VB. 


For a good tutorial on C#, please visit bimlscript.com and click 
the C# Primer link in the Learn Biml Now! lesson. 


Click in the Value textbox of the ReadOnlyVariables property of the 
Script Task Editor, and then click the ellipsis to open the Select Variables 
dialog. Check the checkboxes for the following variables: 


e System::TaskName 

e System::PackageName 

e User::TableCount 

e $Package::IntPkgParam 

e $Package::StringPkgParam 
e $Project::IntProjParam 

e $Project::StringProjParam 


When completed, your Select Variables dialog should appear similar to 
that shown in Figure 2-8. 
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Select Variables E bs 


Select one or more vanables. 


Name Type 
System: :ContainerStart Time DateTime 
System::CreationName String 
System::LocalelD Int32 
System::ParentCortainerGUID String 
System::TaskID String 
System:: Task Name String 
System:: Task TransactionOption Int32 
System: :CancelEvent int 32 
System: Creation Date DateTime 
System: CreatoComputerame String 
System:: ExecutionInstanceGUID String 
System::FailedCanfigurations String 
System::lgnoreConfigurationsOnLoad Boolean 
System::InteractiveMode Boolean 
System::Last Modified Product Version String 
System: Machine Name String 
System: Offline Mode Boolean 
System::Package!D String 
System::Package Name String 
System::Product Version String 
System::-ServerExecution!D int64 
System: Start Time DateTime 
User::TableCount int32 
System:: UserName String 
System: Version Build int32 
System::VersionComments String 
System::VersionGUID String 
System::VersionMajor int32 
System: Version Minor int32 
SPackage::IntPkgParam int32 
$Package::StringfkgP aram String 
$Project::intProjParam Int32 





MOO HOOOOOOROOOROODOOOOOOOOOOROOO00 





| ok || Caneel 








Figure 2-8. The Select Variables dialog 
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Why ProjectName and TaskName? 


As the code below will reveal, you will use these variables to construct a 
value in the subComponent .Net variable in your code. Do you absolutely 
need this variable for SSIS development? No. But you absolutely need this 
variable if you are going to help the Operations team monitor and report 
messages and failures from SSIS logs. For the developer side of DevOps, 
it’s a few lines of code. For the Operations side of DevOps, it surfaces one 
more piece of execution metadata in the logs. Providing this kind and level 
of process instrumentation is vital to the successful integration of DevOps 
and SSIS. 

Click the OK button to close the Select Variables dialog. Your Script 
Task Editor should now appear as shown in Figure 2-9. 


ET Script Task Editor Oo x 


Access Microsoft Visual Studio Tools for Applications (VSTA) to write scripts using the Visual Basic 2015 or Visual C= 2015, 
os) and configure the task's properties. 


oa 
F 






Script 
General 
Expressions 





Se riptLangua ge | Microsoft Visual C# 2015 
EntryPoint Main 
j System::TaskName,System:PackageName,U.... 






EE RRIA 
Figure 2-9. The Script Task Editor 


Click the Edit Script button to open the Visual Studio Tools for 
Applications (VSTA) Editor. Edit the public void Main() method to read as 
follows: 


public void Main() 
{ 


string packageName = Dts.Variables["System: :Package 
Name" ].Value.ToString(); 

string taskName = Dts.Variables| "System: :Task 

Name" |.Value.ToString() ; 
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string subComponent = packageName + + taskName ; 


bool fireAgain = true; 


int tableCount = Convert.ToInt32(Dts.Variables 

[ "User: :TableCount"].Value) ; 

int intPkgParam = Convert. ToInt32(Dts.Variables 
["$Package: : IntPkgParam" |.Value) ; 

string stringPkgParam = Dts.Variables|"$Package: : 
StringPkgParam" |.Value.ToString() ; 

int intProjParam = Convert. ToInt32(Dts.Variables 
["$Project: :IntProjParam"|].Value) ; 

string stringProjParam = Dts.Variables["$Project:: 
StringProjParam" ].Value.ToString(); 


string msg = "Table Count: " + tableCount.ToString(); 
Dts.Events.FireInformation(1001, subComponent, 


msg, "", O, ref fireAgain); 


msg = "Package Parameters: IntPkgParam = " + 


intPkgParam.ToString() + " ; StringPkgParam = " + 


stringPkgParam; 
Dts.Events.FireInformation(1001, subComponent, msg, 


, 0, ref fireAgain); 


msg = "Project Parameters: IntProjParam = " + 


intProjParam.ToString() + " ; StringProjParam = " + 
stringProjParam; 
Dts.Events.FireInformation(1001, subComponent, 


msg, "", O, ref fireAgain); 


Dts.TaskResult = (int)ScriptResults.Success; 
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When completed the code in your VSTA Editor should appear similar 
to that shown in Figure 2-10. 


public void Main() 
{ 


string packageName = Ots.Variables[ “System: :PackageName”].Value.ToString(); 
string taskName = Ots.Variables[ "System: :TaskName" ].Value. ToString(); 
string subComponent = packageName + ".” + taskName; 

bool fireAgain = true; 


int tableCount = Convert. ToInt32(Ots.Variables["User: :TableCount”].Value) ; 

int intPkgParam = Convert. ToInt32(0ts.Variables["$Package: :IntPkgParam"].Value); 
string stringPkgParam = Ots.Variables["SPackage: :StringPkgParam" ].Value.ToString(); 
int intProjParam = Convert. ToInt32(Ots.Variables["$Project: :IntProjParam” ].Value); 
string stringProjParam = Dts.Variables["$Project::StringProjParam" ].Value. ToString(); 


string msg = “Table Count: " + tableCount.ToString(); 
Ots.Events.FireInformation(1001, subComponent, msg, “", ©, ref fireAgain); 


msg = “Package Parameters: IntPkgParam = “ + intPkgParam.ToString() + “ ; StringPkgParam = “ + stringPkgParam; 


Ots.Events.FireInformation(1001, subComponent, msg, “", 8, ref fireAgain); 


msg = “Project Parameters: IntProjParam = “ + intProjParam.ToString() + + stringProjParam; 


DOts.Events.FireInformation(1001, subComponent, msg, “", @, ref fireAgain); 


3 StringProjParam = 


Dts.TaskResult = (int)ScriptResults.Success; 


Figure 2-10. Code in Your VSTA Editor 


Testing .Net Code Compiles Before Closing 
the VSTA Editor 


Have you ever coded away in an SSIS Script task, only to close the VSTA 
Editor and then close the Script Task Editor to see the error shown in 
Figure 2-11? 


cp SQL Get Table Count 


LES SCR Log Values €) 








Figure 2-11. Script task error in VSTA .Net code 
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Kevin Hazzard shared a handy way to test the viability of the .Net code 
contained in the VSTA Editor before closing the editor. Click Build > Build 
ST_... (VSTA scripts are uniquely named), as shown in Figure 2-12. 


DQ] VstaProjects - Microsoft Visual Studio 
File Edit View Project Build | Debug Team Tools Test BimlExpress Analyze Window Help 





oO-O|5-2 EË Run Code Analysis on Solution Alt+F11 s/f 
A PPIE «saa abe oe ; ~ EE ; 
SA scripthisincs > x B 1! Build ST_3f5ac3c0543b488bb8ce2587eecff5ec SS 
7 San: - Fe | Rebuild ST_3fSac3c0543b488bbece2587eectfSec j 
Pam (9) ST_3f5ac3c0543b488bb8 “ould >1_sacsc Piece eet ry fe ST_3fSa 
Š A a ei | 1 MELE 
a 10 z Nan Clean ST_3fSac3c0S43b488bbece2587eechfSec 
x 16 Run Code Analysis on ST_3fSac3cO43b488bboce2587eectfiec 
A 17 =| nar Configuration Manager... 
oll 
2 18 { 
= 19 = /// <summary> 
; /// ScriptMain is the entry point class o e script. 
20 f/f S tM th t t cl f th t 
21 /// or parent of this class. 
22 /// </summary> 





Figure 2-12. Preparing to build a VSTA script 


When you click Build ST_..., the VSTA Editor attempts to build the 
script. In the lower left corner of the screen you will see “Build started,’ as 
shown in Figure 2-13. 


[Build started 
Figure 2-13. Build started 


If there is a bug in the .Net code, the build will fail. If there are no bugs 
in the .Net code, the build will succeed, as shown in Figure 2-14. 





Figure 2-14. Build succeeds or fails 
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Close the VSTA Editor and click the OK button in the Script Task Editor. 
SimplePackage.dtsx should now appear as shown in Figure 2-15. 





Tae wa DataFlow @@ Parameters F] 





op SOL Get Table Count 


SCR Log Values 
£5 servo 


Figure 2-15. SimplePackage.dtsx 


The demonstration SSIS package and project are complete. 


Testing Progress 


‘All software is tested. Some intentionally.” 


- Andy Leonard, circa 2005 


It’s a good idea to always execute your package in the SSIS debugger. 
How else will you know that what you built works? 
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Press the F5 key to start the SSIS debugger. If all goes as planned, you 
should see both tasks succeed, as shown in Figure 2-16. 


3 j 


Package.dtsx [Design] + X 
we Data Flow 





& Parameters E] 





cp SQL Get Table Count 


w 


SCR Log Val 
ES sa uovaues 


Figure 2-16. Success! 


If you click the Progress tab, you should see the OnInformation 


messages raised by your script, as shown in Figure 2-17. 





SimplePackage.dtsx [Design] + X 
ao Control Flow gig DataFlow Wa Parameters [D] EventHandlers ‘s— Package Explorer Cit 
o> 


* validation has started 
=) = Task SCR Log Values 
YO Validation has started (2) 
P Validation is completed (2) 
© Start, 8:12:44m 
O ([SimplePackage.5CR Log Values] Information: Table Count: 13 
O (SimplePackage.SCR Log Values] Information: Package Parameters: IntPkgParam = 42 ; StringPkgParam = Hi There! 
O [SimplePackage.SCR Log Values] Information: Project Parameters: IntProjParam = -99 ; StringProjParam = A project parameter 
© Finished, 8:12:44 PM, Elapsed time: 00:00:00.031 
E > Task SQL Get Table Count 
"P Validation has started (2) 
Validation is completed (2) 
© Start, 8:12:44m 
> Progress: Executing query "Select Count(*) As TableCount From [sys].[tables]". - 100 percent complete 
© Finished, 8:12:44 PM, Elapsed time: 00:00:00. 125 
Validation is completed 
© Start, 8:12:44Pm 
© Finished, 8:12:44 PM, Elapsed time: 00:00:00.266 


Figure 2-17. OnInformation messages 
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Your demo is now ready to begin an epic journey into data integration 


lifecycle management. 


Conclusion 


In this chapter, you built an SSIS package that you will use for the lessons 
throughout this book. I discussed data integration instrumentation and 
messaging to surface SSIS package execution log messages. It’s important 
for us on the Dev side of DevOps to signal supporting personnel on 

the Ops side. It takes minutes to write this code that may save hours of 


troubleshooting. 
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Source Control 


Later in this book, I will discuss why SQL Server database backup and 
restore of the SSISDB database is not a viable method of SSIS code 
promotion. You should back up the SSISDB database just like you should 
back up all databases. You should not back up the SSISDB database for 
code promotion, and you should not back up the SSISDB database in lieu 
of source control. 

There are two types of developers: those who use source control and 
those who will. People ask me, “Which source control engine is best, 
Andy?” My response is, “The one that you use.” Please use source control. 


This section is not an exhaustive tutorial on source control or using 
Team Foundation Services. | do not advocate one source control 
engine over another. 


In this chapter, I will demonstrate using Team Foundation Services 
via Visual Studio Online (visualstudio.com). I will use the terms “source 
control” and “version control” interchangeably because modern source 


control engines also manage software versions. 
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Source Control Client 


Depending on which source control (or version control) engine you desire 
to use, you may or may not require a Visual Studio plug-in (or extension). 
Many developers use Subversion, a popular open-source version control 
system, with Tortoise SVN, a Subversion client implemented as a Windows 
shell. Tortoise is a highly intuitive interface that integrates into Windows 
Explorer to provide context-sensitive menus available by right-clicking 

file system folders that contain project artifacts under Subversion source 
control. 

Git is a popular version control system created by Linus Torvalds (the 
creator of Linux). Git is a distributed source control system, which means 
each developer maintains a local working copy of the repository. Git relies 
heavily on branching (making another - usually local - copy of the current 
version of the code) and merging (the process of adding changes to the - 
usually local - edited version into another version of the code). 

In this chapter I use Team Explorer, the Visual Studio plug-in for Team 
Foundation Services (TFS). You can learn more about Visual Studio plug- 
ins for TFS at visualstudio.com/en-us/docs/tools. 


Creating a Team Project 


In this section, I demonstrate how I use the free source control available 

at Visual Studio Online. I like Visual Studio Online because it’s difficult to 
beat the price (free) and the total cost of ownership (also free). After setting 
up an account you can create a New Team Project from the dashboard. 
Mine is shown in Figure 3-1. 


26 


CHAPTER 3 SOURCE CONTROL 





Visual Studio Team Services Accounts 


v andyleonard.visualstudio.com (Owner) 


Team Projects Actions 
New team project Open in Visual Studio 





Andy Leonard Edit profile 


andyleonard763@hotmail.c 
om 


Microsoft account bd 


© United States 


Figure 3-1. Preparing to create a new team project 


You can think of Visual Studio Online source control as an extension 
of the Visual paradigm. The red box in Figure 3-2 is drawn around an SSIS 
package, which can be considered a Visual Studio project artifact. Project 
artifacts are lowest in the hierarchy of Visual Studio objects. The next 
level is the project, surrounded by a blue box in Figure 3-2. Visual Studio 
projects contain one or more project artifacts. Visual Studio solutions, such 
as the DILMSuite solution circled in green in Figure 3-2, contain one or 


more projects. 


rin nT) OLL Tie = Of aa B nA Dr 

Solution Explorer 7 6 x 
Pa Pa’ F | 

Search Solution Explorer (Ctrl+: P- 


= = 
a) Solution (DILMSamople’ (1 propecti 





te Project.params 
tal Connection Managers 
a te) 55/5 Packages 
a tw) Package Parts 
te! Control Flow 
ta) Miscellaneous 





Figure 3-2. The Visual Studio paradigm 
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Team Foundation Server extends this paradigm an additional level: 
TFS team projects contain one or more Visual Studio solutions. 

Click the “Create team project” link to proceed to the Create New 
Project page, shown in Figure 3-3. 


cy andyleonand 


Projects My favortes biy work items My pull requests 





Create new project 


Projects contain your source code, work items, automated builds and more. 


Project name * 
DILM Community iv 


Description 
Data Integration Lifecycle Management Community Edition projects 


Yersion contral 


Team Foundation Version Control w |D 
Work item process 
Scrum “1 ® 


Figure 3-3. Creating a new team project 


To configure the new team project, add a name and optional 
description, and then select a version control engine (Team Foundation 
Version Control or Git) and a work item process. Click the Create button to 
create the team project and proceed to the Team Project page, as shown in 
Figure 3-4. 
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DIL Community 





DILM Community * Members (1) 
Data Integration Lifecycle Management Community Edition projects A 
x 
; Activity T Days ~ 
/ 
Use continuous integration = 
improve code quality by detecting breaking changes as soon as they happen. #2 
Changesets 
by 1 authors 
Bulld & Release Setup Build 
Lean mort about contmudus mbegratn Mo builds yet 
Wark Add Werk 
No READMEmd found in 2 $/DILM Community = No work items yet 
B 
Add a README 


Help others learn about your project 
Create README 


Leann mare about README files and what to mclude 


Figure 3-4. New team project 


A new team project is now ready to go. 


Configuring SSDT to Use TFS Online 


Returning to SSDT, you next configure a connection to TFS Online. Begin 
by clicking the Team dropdown and clicking Manage Connections, as 


shown in Figure 3-5. 


DQ] DILMSample - Microsoft Visual Studio 
File Edit View Project Build Debug Team | SSIS Tools Test Analyze Window Help 
s 8-2 Ea - -| # Manage Connections... t~| pe 






Gi DataFlow Ķ Parameters [F] EventHandlers "= Package Explorer 


Figure 3-5. Managing source control connections 
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If you haven't configured a connection already, you may first see the 


Connect to Team Foundation Server window shown in Figure 3-6. 


Connect to Team Foundation Server ? x 
Select a Team Foundation Server: 
Team Project Collections: Team Projects: 
com (Switch User) Connect Cancel 


Figure 3-6. Connecting to Team Foundation Server 


Click the Servers button to open the Add/Remove Team Foundation 
Server dialog, as shown in Figure 3-7. 
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Add/Remove Team Foundation Server ? x 


Team Foundation Server list: 


Name URL Add... 








Figure 3-7. Add/Remove Team Foundation Server 


Click the Add button to open the Add Team Foundation Server dialog, 
as shown in Figure 3-8. 


Add Team Foundation Server ? x 


Name or URL of Team Foundation Server: 





Preview: | https://andyleonard.visualstudio.com/ 














Figure 3-8. Adding a Team Foundation Server 
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Enter the URL of your TFS Online account page in the “Name or URL of 
Team Foundation Server” textbox. Click the OK button to proceed. 

The Add Team Foundation Server dialog closes and the Add/Remove 
Team Foundation Server dialog should now list your TFS Online 


configuration and appear similar to that shown in Figure 3-9. 


Add/Remove Team Foundation Server ? x 


Team Foundation Server list: 


Name URL Add... 


andyleonard.visualstudio.com https://andyleonard.visualstudio.com/ 


Close 


Figure 3-9. Configured Add/Remove Team Foundation Server 
dialog 


Click the Close button to return to the Connect to Team Foundation 
Server dialog. 
When the Connect to Team Foundation Server dialog displays, select 


your new team project, as shown in Figure 3-10. 
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Connect to Team Foundation Server ? x 


Select a Team Foundation Server: 


andyleonard.visualstudio.com v 


Team Project Collections: 


sa andyleonard 
M] DILM Community 





com (Switch User) | ‘Connect | | Cancel 


Figure 3-10. Selecting a team project 


Click the Connect button. The Connect page of Team Explorer should 


now appear, similar to that shown in Figure 3-11. 





a paai, ' 
| onne 


(EE A # | © Search Work Items (Ctrl+') p~ 


Connect | DILM Community 





Manage Connections ~ 


4 andyleonard.visualstudio.com (1) 
zg, DILM Community 


4 Local Git Repositories 
New ~» | Add ~» | Clone ~» | View Options ~ 


Add or clone a Git repository to get started. 
Figure 3-11. Team Explorer connected to TFS Online 
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In Solution Explorer, right-click the project name (DILMSample), 
hover over Source Control, and then click Add Project to Source Control, as 


shown in Figure 3-12. 





Search Solution Explorer (Ctri+:) 





i} DILMSample 
g Project.params Deploy 


EA SimplePackage.dtsx cae 
a fal Package Parts New Solution Explorer View 
fal Control Flow Add > 
fe) Miscellaneous Manage NuGet Packages... 
Debug + 
Convert to Package Deployment Model 
Source Control » ti Add Project to Source Control... 
a Cut Ctrl+X 
Paste Ctrl+V 
C Rename 
# = Properties Alte Enter 


Figure 3-12. Preparing to add the project to source control 


The Add Solution <Solution Name> to Source Control dialog displays 
and contains the name of your team project in the team Project Locations 


listbox, as shown in Figure 3-13. 
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Add Solution DILMSample to Source Control f X 
Indicate where to store your solution and projects in the Team Foundation Server and in your 
workspace. 

Team Foundation Server Details 


Server; andyleonard.visualstudio.com\andyleonard 


Team Project Location: 
v mf DILM Community 
» E BuildProcessTemplates 





Make New Folder 


Type a name for the solution folder: 
DILMSample 




















Solution and project files will be added to: 


|$/DILM Community/DILMSample Advanced... 


Workspace: 
VMDEMO v 


Figure 3-13. Adding a solution to source control 


Click the OK button to add the DILMSample project to source control. 

Once added, the solution artifacts are decorated with + symbols in 
Solution Explorer. The + symbol indicates the file is “newly added” to 
source control but not yet checked in, as shown in Figure 3-14. 
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+E DILMSample 
+@@ Project.params 
fa) Connection Managers 
4 tm) SSIS Packages 
+E, SimplePackage.dtsx 






Newly added file 
Control Flow 
fa) Miscellaneous 





Figure 3-14. Newly added to source control 


Check in the solution by right-clicking the project name, hovering over 
Source Control, and clicking Check In, as shown in Figure 3-15. 





= 7 
Solution Explorer 


Li 


©°M|o-Ssa@als— 


Search Solution Explorer (Ctrl+;) 


ow) DILMSample 





+@@ Project.params | Deploy 
im] Connection Managers | y 


Build 
a ta) 5515 Packages 
+E, SimplePackage.dtsx Rebuild 
a tml Package Parts New Solution Explorer View 
Control Flo 
ta] Control Flow Ra : 


fal Miscellaneous 
Manage NuGet Packages... 


Debug + 
Convert to Package Deployment Model 
Source Control r | “Check Ins 
db Cut Ctrl+X | ") Undo Pending Changes... 
Paste Ctrl+V | + 


co Shelve Pending Changes 


Figure 3-15. Checking in the solution 


The Team Explorer Pending Changes window displays. It is a best 
practice to always add a version comment, as shown in Figure 3-16. 
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| G © a F | G Search Work Items (Ctrl+") 


ee a oe ie p a f henna 
eami CxpIiorer - Pending Changes 
=o =n = J `= 7 


Pending Changes | DILM Community 


VMDEMO + 
Checkin Shelve + | Actions + 


4 Comment 
20170806 - Andy Leonard - initial checkin] 


4 Related Work Items 
Queries ~ | Add Work Item by ID + 


Drag work items here to link them to the check-in. 


4 Included Changes (5) 
Exclude All | View Options ~ 
4 fe! E:\Projects\DILMSample 
4 i=! DILMSample 
DILMSample.database [add] 
DILMSample.dtproj [add] 
D Project.params [add] 
=) SimplePackage.dtsx [add] 
Wd DILMSample.sin [add] 


4 Excluded Changes (1) 
Include All | View Options ~ | Detected: 4 add(s), 3 delete(s) 


4 (ea! E:\Projects\IESSIS1_May2017_Biml 
&l deploy [add] 


Figure 3-16. Preparing to check in the project 





You may or may not be prompted to confirm the check-in, as shown in 


Figure 3-17. 


37 


CHAPTER 3 SOURCE CONTROL 


Check-in Confirmation 


As — Continue to check-in 5 items? 


E 
Don't prompt again 


Figure 3-17. Check-in confirmation 


Once checked in, Team Explorer’s Pending Changes page will appear 


as shown in Figure 3-18. 






pA a eee Sac Se aoe 
@am cxplorer Mending ¢ ranges 
E e —_ nu _ — — — = i — — a 








S C seach wort tere cut 5 
Pending Changes | DILM Community -|7 

 @ Changeset 989 successfully checked in. X 
VMDEMO ~ 


Check In Shelve + | Actions + 
4 Comment 
Enter a check-in comment 


4 Related Work Items 
Queries ~ | Add Work Item by ID + 


Drag work items here to link them to the check-in. 


4 Included Changes 
Exclude All | View Options + 


Drag changes here to include in the check-in. 


4 Excluded Changes (1) 
Include All | View Options ~ | Detected: 4 add(s), 3 delete(s) 


4 Gal E:\Projects\IESSIS1_May2017_Biml 
| deploy [add] 


Figure 3-18. All checked in 


38 


CHAPTER 3 SOURCE CONTROL 


Once checked in, Solution Explorer will display lock decorations 
beside checked-in artifacts, as shown in Figure 3-19. 





Search Solut epDrorer f 
â Project.params 
fa) Connection Managers 
a tm) SSIS Packages 
SE, SimplePackage.dtsx 
a t) Package Parts 
fa) Control Flow 
fa) Miscellaneous 


Figure 3-19. Checked-in solution artifacts 


In this chapter, I demonstrated how to use Team Foundation Services 
(TFS) Online via the Visual Studio Online website. For more information, 
please visit visualstudio.com. 
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Deploy to the SSIS 
Catalog 


The SSIS catalog was released with SSIS 2012. Each release of SSIS since 
2012 has featured upgrades to catalog functionality. There are several ways 
to deploy an SSIS project to the SSIS catalog. In this chapter, I will present 
two deployment options: 


e Deploy from SSDT 
e Deploy from an ISPAC file 


A key tenet of DevOps is automation, and in this chapter I will discuss 


and demonstrate automated SSIS deployments. 


Deploying from SSDT 


Since the first release of the SSIS catalog, SSIS has been backwards- 
compatible. Prior to 2012, the deployment model for SSIS had no name; 
there was only one way to deploy SSIS and three targets: the file system, the 
MSDB database, and the “SSIS Package Store” (which was, by default, one 
folder in the file system and the MSDB database combined). Deployment 
to the SSIS catalog is a new way to deploy SSIS packages, so we needed a 
name for the “old way” as well as a name for the “new way.’ The new way 
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is called “project deployment model” and the old way is called “package 
deployment model.” 

In this chapter (and book), I focus on the project deployment model. 
The project deployment model is required to deploy SSIS packages to the 
SSIS catalog. 

To deploy your demo project, open the project in SSDT, right-click the 
project name in Solution Explorer, and click Deploy, as shown in Figure 4-1. 





2M\e-s@a\s—= 


Search Solution Explorer (Ctrl+;) 





| 3 DILMSample 
a@@ Project.params Deploy 
f) Connection Managers oh Build 
4 ta) SSIS Packages 
a SimplePackage.dtsx Rebuild 
a ġa] Package Parts 6! New Solution Explorer View 
al Control Flow Add 
ta) Miscellaneous 
Manage NuGet Packages... 


Figure 4-1. Deploying the DILMSuite SSIS project from SSDT 


The Integration Services Deployment Wizard is an extremely functional 
piece of software. The underlying executable, ISDeploymentWizard. exe, 
sports a graphical user interface (GUI) and a command-line interface (CLI). 
The utility supports deployment from ISPAC file to SSIS catalog as well as 
SSIS catalog to SSIS catalog and, starting with SSIS 2016, supports package 
deployment as well as project deployment. I won't dive deeply into this 
awesome application; just know that I truly appreciate the excellence 
apparent in the Integration Services Deployment Wizard. 

The Integration Services Deployment Wizard displays the Introduction 
page or the Select Destination page (if you previously checked the “Do not 
show this page again” checkbox on the Introduction page), as shown in 
Figure 4-2. 
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Aj’ Integration Services Deployment Wizard — m x 


d : 
f. À Introduction 





Select Source 
Deploy Integration Services project or packages. 
Select Destination 
Review This wizard deploys Integration Services project or packages to an Integration Services catalog on an 
i instance of SOL Server. 
Results 
There are five steps to completing this wizard: 


1, Select the Deployment Model: Project Deployment or Package Deployment. 
2. Select the Integration Services project or packages thal you want to deploy. 
3. Select the destination. 

4. Review your selections. 

5. Deploy the project or packages. 








Figure 4-2. The Integration Services Deployment Wizard 
Introduction page 


If the Introduction page is displayed, click the Next button to proceed 


to the Select Destination page. 


Did | skip over the Select Source page? Yes. When started from inside 
SSDT, the Integration Services Deployment Wizard already has the 
information it needs about the source. 


Enter the name of server that hosts your SSIS catalog. Click the Browse 
button to select the target catalog folder. When completed your Select 
Destination page should appear similar to that shown in Figure 4-3. 
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Aj Integration Services Deployment Wizard = o x | 


Ed 
Ai n Select Destination 


Introduction G) Help 
Select Source 


Enter the destination server name and where the project will be located in the 
Integration Services catalog. 


Server name: 
vmDeme Demo Browse... 
Path: 


/SSISDB/Test/DILMSample 














SSIS Catalog CatalogFolder SSIS Project 


€ Previous Next > Depto Cancel 


Figure 4-3. The Integration Services Deployment Wizard Select 
Destination page 


The Path value shown on the Select Destination page refers to the SSIS 
catalog path of the SSIS project. SSIS catalog projects are contained in an 
SSIS catalog folder, which is contained in an SSIS catalog (which is always 
named SSISDB). SSIS packages are contained in SSIS catalog projects. 

Click the Next button to proceed to the Review page, as shown in 
Figure 4-4. 
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AF Integration Services Deployment Wizard - m x 
DÝ E 
Pah Review 
H 
Introduction Q Hep 
z Review your selections. 
Select Destination 
C oor 
Results fe Use the following arguments to perform this deployment from the command ine: 





deo | Copy Cec | 
EVEIOp Mer UUR 


Path: E:\Projects\DILMSample\DILMSanipie art 
Destinati 


Server name: vmDemo\Demo 
Path: /SSISDB/Test/DILMSample 








< Previous å Deploy Cancel 


Figure 4-4. Reviewing deployment selections and options 


It’s easy to skip right past the Integration Services Deployment Wizard 
Review page without giving it a second glance, but I want you to pause 
here with me for a moment because there is some information here that 


may prove useful to you in the future (or present). 


Deploying from the Command Line 


Deployment via CLI is powerful and useful for DevOps automation. The 
Source and Destination information is self-explanatory. Please note in 
the screenshot that I right-clicked the Command line to surface a context 


menu with a single option: Copy. I’ve pasted the command line here: 
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Command line: /Silent /ModelType:Project /SourcePath: "E:\ 
Projects\DILMSample\DILMSample\bin\Development\DILMSample. 
ispac’ /DestinationServer: “vmDemo\Demo" /DestinationPath: 
"/SSTSDB/Test/DILMSample" 


SSIS ships with a number of utilities, some of which may be called 
from a command prompt. Any utility that can be called from a command 
prompt may be scheduled using SQL Agent (provided command-line 
execution is enabled), Windows Scheduler, or almost any other scheduling 
service or utility. 

In order to use the command line shared on the Integration Services 
Deployment Wizard Review page, you need to know a couple things: 


1. The location of the executable 
2. The meaning of the switches and values 


I installed the Demo instance of SQL Server on vmDemo’s E: drive so 
the location of ISDeploymentWizard.exe is E:\Program Files\Microsoft 
SOL Server\130\DTS\Binn\ISDeploymentWizard.exe, as shown in 
Figure 4-5. 


B! 0E- Application Tools Binn 
Home Share View Manage 
= v A [d > ThisPC > vmDemo_E (E:) > Program Files > Microsoft SQL Server > 130 > DTS > Binn 


Name 
ge Quick access 
AŤ lSDeploymentWizard.exe 


E This PC YA |SDeploymentWizard.exe.config 
Ml Desktop AF \SProjectWizard.exe 


D Docin "h ISProjectWizard.exe.config 


JẸ Downloads 


[EF] ISServerExec.exe 


YA ISServerExec.exe.config 


Figure 4-5. The location of the IsDeploymentWizard.exe file 


This is the same application that is currently open and executing; it is 
the Integration Services Deployment Wizard executable. As mentioned 
earlier, the executable exposes two interfaces: a GUI and a CLI. It’s possible 
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to execute the entire deployment of your SSIS project, DILMSuite, by 
opening a command prompt, supplying the path to ISDeploymentWizard. 
exe followed by the command line above. Don’t believe me? Check out 
Figure 4-6. 


EE Command Prompt - o x 


E:\>"E:\Program Files\Microsoft SQL Server\130\DTS\B8inn\ISOeploymentWizard.exe” /Silent /ModelType:Project /SourcePath:” 
E:\Projects\DILMSample\DILMSample\bin\Development \DILMSample.ispac” /DestinationServer: “vaDemo\Demo” /DestinationPath:”/ 
SSISOB/Test/DILMSample” 





Figure 4-6. Deployment via the command line 


There’s good news and bad news here. The good news is this execution 
succeeded. The bad news? There is no feedback provided if the execution 
succeeds. How can you tell it succeeded? Well, it didn’t fail. What does a 
failure look like? A failure may result in a prompt from the CLI that gives 
you a list of the acceptable switches, as shown in Figure 4-7. 


SOL Server Integration Services x 


/Silent[+|-] Default value: (short form /5) 
/SourceType:{File|Server} Default value: File’ (short form /5T) 
/ModelType:{Project|Package} Default value: Project’ (short form /MT) 


a 
k a 





/SourcePath: <string> (short form /SP) 
/SourceServer:<string> (short form /SS) 
/Packages:<string> (short form /PK) 


/ProjectPassword:<string> (short form /PP) 
/DestinationServer:<string> (short form /DS) 
/DestinationPath:<string> (short form /DP) 
/SkipSource[+|-] Default value:'" 

@<file> Read response file for more options 


is 


Figure 4-7. Deployment failure using the CLI 


This isn’t the only failure response. For example, if you mistype the 
name of the destination server, you will get a “Failed to connect” message 
after ISDeploymentWizard.exe tries to contact the server for 30 seconds. 
That message will appear similar to Figure 4-8. 
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SQL Server Integration Services x | 


(x) Failed to connect to server vmDemo\Dem. (Microsoft.SqlServer.Connectioninfo) 
Additional information: 
L» Anetwork-related or instance-specific error occurred while establishing a connection to SQL 
Server. The server was not found or was not accessible. Verify that the instance name is correct 


and that SQL Server is configured to allow remote connections. (provider: SQL Network 
Interfaces, error: 26 - Error Locating Server/Instance Specified) (Microsoft SQL Server) 


TT 
Figure 4-8. Deployment failure due to connection error 


Another way to detect success is to use SSMS to connect to the SSIS 
catalog and check for the existence of your SSIS project, as shown in 
Figure 4-9. 


3 & Integration Services Catalogs 
3 (Ay ssisoB 
ə) & Framework 
=> @& Test 
3 E Projects 


Figure 4-9. The SSIS project deployed to the SSIS catalog 


If the SSIS project has been deployed before, though, you can check 
by right-clicking the SSIS project in the catalog and clicking Versions, as 
shown in Figure 4-10. 


=) © Integration Services Catalogs 
= [A SSISOB 
& 0 Framework 
=) Me Test 
© © Projects 
2 SPER 
& A Environments Configure... 
& 3 SOL Server Agent 


Validate... 





Figure 4-10. Opening SSIS catalog project versions 
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The Project Versions dialog displays as shown in Figure 4-11. 





Sy Project Versions = oO x 
| Select a page LT Script ~ | @) Help 
# Versions 
Versions: 
| Curent D | Time Las Renod Description Project LSN ¥ 
| (| eve2or7 1:20.07PM | (19,018 
| Lj Bb 201 7 $1629 AM 10.018 


Figure 4-11. Viewing SSIS catalog project versions 


The last deployment is marked as Current. 

Returning to your deployment from SSDT, if you click the Deploy 
button on the Integration Services Deployment Wizard Review page, the 
Deployment Wizard will attempt to deploy your SSIS project to the SSIS 
catalog you selected, into the SSIS catalog folder you specified, as shown in 
Figure 4-12. 
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AY Integration Services Deployment Wizard —_ m x 
_ 
Pie Results 


introduction 


Select Source R its 
Select Destination 


Review 


O| Loading project 
©) Connecting to destination server 


Results 





Save Report... 


Figure 4-12. Successful deployment 


If all goes as hoped, the deployment will be a success. 


Deployment Failures 


Sometimes bad things happen to good deployments. You may be reading 
this book because you experienced a deployment failure triggered by a 
good-faith effort to restore the SSIS catalog database, SSISDB. 


Backup and restore is not a recommended method for promoting or 
transferring SSIS projects between SSIS catalogs! 
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When you attempt to deploy an SSIS project (or package) to an SSIS 
catalog after SSISDB was improperly restored, the deployment will not 
succeed. Instead, your Integration Services Deployment Wizard Results 


page will appear as shown in Figure 4-13. 


lar ntegration Services Deployment V 


a i 
fi ù Results 


Introduction w Help 


Select Source Results 
Select Destination 
Review 


Results 


SOL Server Integration Services 


gP, Please ceate a master key in the database or open the master key in the session before performing this 


p operation. (Microsoft SQL Server, Error: 15531) 





Figure 4-13. Failed SSIS project deployment 


When you click the Failed link, the message box in Figure 4-13 will 


display the error message, 


Please create a master key in the database or open 
the master key in the session before performing this 
operation. (Microsoft SQL Server, Error: 15581) 


Please find more information about backing up and restoring the SSIS 
catalog database (SSISDB) on the SSIS Catalog page in the section titled 
“Back up, Restore, and Move the SSIS Catalog” (docs.microsoft.com/ 
en-us/sql/integration-services/service/ssis-catalog). I blogged 
about this scenario and the post includes a script one can use to restore 
SSISDB. You can find my post titled “Deploying SSIS Projects to a Restored 
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SSIS Catalog” at andyleonard.blog/2017/07/29/deploying-ssis- 
projects-to-a-restored-ssis-catalog-ssisdb. The script I use is the 
following: 


ha 


My script for restoring SSISDB to a SQL Server 2016 SP1 
instance of SOL Server. 
I followed the instructions found at https://msdn.microsoft. 
com/en-us/library/hh213291(v=sql.130).aspx 


Hope this helps, 
Andy Leonard 
**K Action is required where you see three asterisks °***" 


y 


-- create the ##MS SSISServerCleanupJobLogin## login if it does 
not already exist. 

USE [master | 

GO 


print ‘##MS SSTSServerCleanupJobLogin## login’ 

If Not Exists(Select [name] 

From sys.sql logins 

Where [name] = '##MS_ SSTSServerCleanupJobLogin##' ) 

begin 

print ' - Creating the ##MS SSISServerCleanupJobLogin## login’ 
CREATE LOGIN [##MS_ SSISServerCleanupJobLogin##] WITH PASSWORD 
=N' DWehrJfiRgMxEFaE=KxomUkF7fnV3poW/ZOPJ' -- *** change this, 
please - Andy 

, DEFAULT DATABASE=[master | 

, DEFAULT LANGUAGE=[us english] 

» CHECK EXPIRATION=OFF 

» CHECK POLICY=OFF 
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print ' - ##MS SSTSServerCleanupJobLogin## login created’ 
end 
Else 
print 
GO 


- ##MS_SSISServerCleanupJobLogin## already exists. ' 


print 


print ' - Disabling the ##MS SSISServerCleanupJobLogin## login’ 
ALTER LOGIN [##MS_SSISServerCleanupJobLogin##] DISABLE 

print ' - ##MS_SSISServerCleanupJobLogin## login disabled' 

GO 


USE [master] 
GO 


SET ANSI NULLS ON 
GO 


SET QUOTED IDENTIFIER ON 
GO 


print ‘dbo.sp ssis startup stored procedure’ 
If Exists(Select s.name + '.' + p.name 

From sysS.procedures p 

Join sys.schemas s 

On s.[schema_ id] = p.[schema id] 


Where s.[name] = ‘dbo' 

And p.name = ‘sp ssis startup') 

begin 

print ' - Dropping dbo.sp ssis startup stored procedure’ 
Drop PROCEDURE [dbo].[sp ssis startup] 

print ' - dbo.sp ssis startup stored procedure dropped’ 
end 
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print ' - Creating dbo.sp ssis startup stored procedure’ 
go 


CREATE PROCEDURE [dbo].[sp ssis_startup] 
AS 
SET NOCOUNT ON 
/* Currently, the IS Store name is 'SSISDB' */ 
IF DB ID('SSISDB') IS NULL 
RETURN 


IF NOT EXISTS(SELECT name FROM [SSISDB].sys.procedures WHERE 
name=N' startup’ ) 
RETURN 


/*Invoke the procedure in SSISDB */ 

/* Use dynamic sql to handle AlwaysOn non-readable mode*/ 
DECLARE @script nvarchar(500) 

SET @script = N'EXEC [SSISDB].[catalog].[startup |’ 
EXECUTE sp executesql @script 

GO 
print 


- dbo.sp ssis startup stored procedure created’ 


print 


use master 

go 

print ‘Enabling SOLCLR' 

exec sp configure ‘clr enabled’, 1 
reconfigure 

print ‘SQLCLR enabled’ 

print '' 

print ‘MS SQLEnableSystemAssemblyLoadingKey asymetric key’ 
If Not Exists(Select [name] 

From sys.asymmetric_ keys 
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Where [name] = 'MS SQLEnableSystemAssemblyLoadingKey' ) 
begin 


print ' - Creating MS SQLEnableSystemAssemblyLoadingKey ' 
Create Asymmetric key MS SQLEnableSystemAssemblyLoadingKey 
From Executable File = ‘E:\Program Files\Microsoft SQL 
Server \130\DTS\Binn\Microsoft.SqlServer.IntegrationServices. 
Server.dll' -- *** check this, please - Andy 


print ' - MS SQLEnableSystemAssemblyLoadingKey created’ 
end 
Else 
print 
go 
print 


- MS _SQLEnableSystemAssemblyLoadingKey already exists. ' 


print ‘MS SQLEnableSystemAssemblyLoadingUser SQL Login' 
If Not Exists(Select [name] 

From sys.sql logins 

Where [name] = 'MS SQLEnableSystemAssemblyLoadingUser' ) 
begin 


print ` - Attempting to create MS_ 
SQLEnableSystemAssemblyLoadingUser Sql login’ 

begin try 

Create Login MS SQLEnableSystemAssemblyLoadingUser 

From Asymmetric key MS SQLEnableSystemAssemblyLoadingKey 


print ' - MS SQLEnableSystemAssemblyLoadingUser Sql login 
created’ 
print ' - Granting Unsafe Assembly permission to MS SQL 


EnableSystemAssemblyLoadingUser'' 
Grant unsafe Assembly to MS SQLEnableSystemAssemblyLoadingUser 


print ' - MS SQLEnableSystemAssemblyLoadingUser granted Unsafe 
Assembly permission’ 


end try 
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begin catch 

print ' - Something went wrong while attempting to create the 
MS SQLEnableSystemAssemblyLoadingUser Sql login, but it''s 
probably ok...’ 

-- nothing for now 

end catch 

end 
Else 
print 


- MS _SQLEnableSystemAssemblyLoadingUser Sql login 
already exists. ' 


go 


print 


print 'Restoring SSISDB' 
USE [master] 


begin try 

ALTER DATABASE [SSISDB] SET SINGLE USER WITH ROLLBACK IMMEDIATE 
end try 

begin catch 

-- ignore the error (usually happens because the database 
doesn't exist...) 

end catch 


RESTORE DATABASE [SSISDB] 

FROM DISK = N'E:\Andy\backup\SSISDB_SP1.bak' -- *** check 
this, please - Andy 

WITH FILE = 1, 

MOVE N'data' To N'E:\Program Files\Microsoft SQL Server\ 
MSSQL13.TEST\MSSQL\DATA\SSISDB.mdf', -- *** check this, 
please - Andy 
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MOVE N' log’ TO N'E:\Program Files\Microsoft SQL Server\MSSQL13. 
TEST\MSSQL\DATA\SSISDB.1df', -- *** check this, please - Andy 
NOUNLOAD 

, REPLACE 

, STATS = 5 


ALTER DATABASE [SSISDB] SET MULTI USER 


GO 
print 


- SSISDB restore complete’ 


print '' 

print ‘Set ProcOption to 1 for dbo.sp ssis startup stored 
procedure ' 

EXEC sp procoption N'[dbo].[sp ssis startup]', ‘startup’, '1' 
print ‘ProcOption set to 1 for dbo.sp ssis startup stored 
procedure ' 


GO 
print 


Use SSISDB 
go 


print '##MS_SSISServerCleanupJobUser## user in SSISDB database' 
If Not Exists(Select * 

From sys.sysusers 

Where [name] = '##MS_ SSISServerCleanupJobUser##' ) 

begin 
print ' - Creating ##MS SSTSServerCleanupJobUser## user’ 
CREATE USER [##MS_ SSTSServerCleanupJobUser##] FOR LOGIN 
[##MS SSTSServerCleanupJobLogin##] WITH DEFAULT SCHEMA=| dbo | 
print ' - ##MS SSTSServerCleanupJobUser## user created’ 


end 


of 
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Else 

print ' - ##MS_SSISServerCleanupJobUser## already exists. ' 
GO 
print ~ 


/* 


-- One method for restoring the master key from the file. 
-- NOTE: You must have the original SSISDB encryption password! 


Restore master key from file = ‘E:\Andy\backup\SSISDB SP1_ 


key ' -- *** check this, please - Andy 

Decryption by password = 'SuperSecretPassword' -- 'Password 
used to encrypt the master key during SSISDB backup' -- KK 
check this, please - Andy 

Encryption by password = 'SuperSecretPassword' -- ‘New 
Password ' -- *** check this, please - Andy 

Force 

go 

a7] 


-- Another method for restoring the master key from the file. 

-- NOTE: You must have the original SSISDB encryption password! 
print ‘Opening the master key’ 

Open master key decryption by password = ‘SuperSecretPassword' 
-- ‘Password used when creating SSISDB' = -- *** check this, 
please - Andy 

Alter Master Key 

Add encryption by Service Master Key 

go 

print ‘Master key opened' 


print 
print ‘Checking the SSIS Catalog Schema Version’ 
exec [catalog].check schema version @use32bitruntime = 0 
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The Transact-SQL script listed above requires you to supply 
the password used to create the SSIS catalog; replace the text 
“SuperSecretPassword” with the password for your SSIS catalog. Each 
place you see the comment, -- *** check this, please - Andy, please 
check the preceding line of T-SQL. 


Conclusion 


In this chapter, I demonstrated deploying an SSIS project to the SSIS 
catalog. You examined the Integration Services Deployment Wizard’s GUI 
and CLI. I discussed the importance of deployment in any enterprise that 
practices DevOps. 
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Configure the SSIS 
Catalog Project 


A key tenet of enterprise DevOps is externalization, which is the 
configuration or parameterization of code. The configuration is stored 
apart from the code itself. Externalization promotes code reuse and 
supports decoupling, both best practices in software development. In this 
chapter, I focus on externalizing SSIS connection manager connection 
string properties because I believe it is an SSIS best practice to manage 
SSIS project and package connection strings via external configurations. 

Your DILMSample SSIS project is deployed to your SSIS catalog. 
Expanding the Integration Services Catalogs node in SSMS’s Object 
Explorer surfaces the view shown in Figure 5-1. 


= & Integration Services Catalogs 
3 [A SSISDB 
4) ©) Framework 
= S Test 
=) E Projects 
=) (5) DILMSample 
=) m Packages 
=), SimplePackage.dtsx 
4) ©) Environments 


Figure 5-1. The DILMSample SSIS project in the Integration Services 
Catalogs node, as shown by SSMS’s Object Explorer 
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You see the SSIS catalog (SSISDB), two SSIS catalog folders 
(Framework and Test), one SSIS catalog project in the Test catalog folder 
(DILMSample), and one SSIS package in the DILMSample catalog project 
(SimplePackage.dtsx). 

Why do I write SSIS project in some places and SSIS catalog project in 
other places? I am referring to two distinct objects. 


1. An SSIS project exists in the context of SQL Server 
Development Tools (SSDT). It is a (hopefully source- 
controlled) project under development. 


2. An SSIS catalog project is an SSIS project that has 
been deployed to an SSIS catalog. 


It will help to think of SSIS catalog projects as separate and distinct 
objects, separated from the SSIS projects that exist outside the SSIS 
catalog, especially when thinking of SSIS catalog project configuration. The 
two are weakly coupled in this fashion: changes made to the SSIS project 
may impact its related SSIS catalog project, and then only after deployment 
to the SSIS catalog. Changes made to the SSIS catalog project in no way 
impact the SSIS project. 


Configuring Projects 


You have several options for configuring your SSIS catalog project. In SSMS 
Object Explorer’s Integration Services Catalogs node, expand the subnodes 
to the DILMSample project level. Right-click the DILMSample SSIS catalog 
project and click Configure, as shown in Figure 5-2. 
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3 & Integration Services Catalogs 








3 [A ssisoB 
4 & Framework 
= S Test 
= E Projects 
; et DILMSam 
Ð @ Environments | Configure... 
D J SQL Server Agent aa 
Move... 
Versions... 


Figure 5-2. Preparing to configure the DILMSample catalog 
project 


The Configure - DILMSample dialog opens. 


Configuring Connections 


The SSIS catalog treats connection manager properties like parameters. 
In fact, if you query the [catalog ].[ object parameters | view, you will 
find parameters with names like “CM.<SSIS Connection Manager Name>. 
ConnectionString’ This is a connection manager parameter. I’m pretty 
sure “CM” indicates “connection manager.’ “<SSIS Connection Manager 
Name>.ConnectionString” is a connection manager’s ConnectionString 
property, for sure. 

When the Configure - DILMSample window displays, click the 
Connection Managers tab to view the connection’s configuration, as 


shown in Figure 5-3. 
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By Configure - DA MSampie = 0 4 


| Selec a page C Script = 0 Hap 








| Propexty rhe - Vaie 
Date Soucernm Demo Deno inta Catalogs Tea D8 integrated Securty Tre Appicaton Hames. 
(rstas Tee DB | 
Tamoin [Paene | 
p" vnema Deno NWDEMOA | RetsinSemeCannection | False 
Serrerilara [mbeto Dene 
LsorMore ULL 
Mirean pment 
Progress 
üi 
mae 


Figure 5-3. Viewing the connection manager's configuration 


Note the listbox below the Parameters and Connection Managers 
tabs. It surfaces the container (a catalog project artifact) and name (of 
the connection manager). My SSIS project contains a single connection 
manager in SimplePackage.dtsx, so SimplePackage.dtsx is the container 
and the connection manager name is vmDemo\Demo. TestDB1. The name 
of your connection manager will almost certainly differ, as will your 
ConnectionString property, to which you next turn your attention. 

The value in the ConnectionString property is the value configured 
at design time. The SSIS catalog will always have access to design-time 
default values. If I execute the SimplePackage. dtsx SSIS package in 
the SSIS catalog as it is currently configured, it will execute and attempt 
to connect to the vmDemo\Demo instance of SQL Server and the TestDB 
database. Because I've deployed this SSIS project to the SSIS catalog 
hosted on the vmDemo\Demo SQL Server instance, the SimplePackage.dtsx 
package should execute without issue. 

But what happens when I deploy the DILMSample SSIS project to 
a different SSIS catalog? Simp lePackage.dtsx may or may not execute 
successfully, depending on a number of factors that impact whether 
processes executing on one server can connect to SQL Server instances 


hosted on other servers. 
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One key thing to remember is this: as the SSIS project is configured 
at this time, each time I deploy the DILMSample SSIS project, the 
default value of the vmDemo\Demo. TestDB1 connection manager in 
SimplePackage.dtsx will always be configured to connect to the vmDemo\ 
Demo instance of SQL Server and the TestDB database on that instance. 


Overriding the Connection Configuration 


How does one change the connection configuration? Click the ellipsis 
to the right of the ConnectionString property’s Value textbox, circled in 
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Figure 5-4. Preparing to open the ConnectionString’s Value 
configuration 


The Set Parameter Value dialog opens. 


In the SSIS catalog, project parameters, package parameters, and 
connection manager properties are all considered parameters. 
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Select and copy the design-time default value of the ConnectionString 
property, found in the textbox beside the “Use default value from package” 


option, and paste it into the “Edit value” textbox, as shown in Figure 5-5. 


ls Set Parameter Value 


Parameter: 





[Simple Package dtax] [CM vmDemo\Demo.TestDB1 ConnectionString] 





Data Source=focal\Demo;lnitial Catalog=TestDB: Integrated Securty=True Application 
@ Edt value: Name=SSIS-SimplePackage-vmDemo\Demo. TestDB1: 





©) Use default value from package | Data Source=vmDemo\Demo:Initial Catalog=TestDB: integrated Securty=True:Application Name 





[ox || cores || Hee 
Figure 5-5. Editing the ConnectionString property value 


I made a couple changes to the ConnectionString after pasting it into 
the Edit Value textbox. I deleted a Guid value from the Application Name 
setting and I changed the data source from vmDemo\Demo to (local) \Demo. 
The data source is technically the same as before. I made the change 
so that I can later be certain which value is being used: the design-time 
default or this literal override. 


Click the OK button to proceed to the override page in Figure 5-6. 
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ih Configure - DILMSample a 2) 7 


| Select a page {J Script ~ | @ Help 














Conlara Mame tee 





| Connection String Data Source=(local)\ Demo: initial Catalog=Teat DB:integrated...[.. | 
[PiiaiCatalog TestDB =| 
eaman [Password a [ 
i owmDemo Demo (VMDEMO'A, | j 
T Fay Leonard] om | Retain SameConnection | False Ibe | 
[ServerName vm Demo" Deng Si | 
| Userame NULI lad 
m : : 
Progress: 


Cancel Help 
Figure 5-6. A catalog project literal override 


The ConnectionString value has been overridden by an SSIS catalog 
literal override. This is indicated by the bold text decoration of the value. 
You can think of a literal override as hard-coding a configuration value 
into the SSIS catalog. Subsequent deployments of the DILMSample SSIS 
project to this SSIS catalog will not change the value of the literal override, 
even if you change the value of the connection manager’s design-time 
default ConnectionString value. This is one reason I refer to an SSIS project 
in SSDT and an SSIS catalog project or catalog project once it has been 
deployed to an SSIS catalog. 

As someone with permission to configure an SSIS catalog, you can 
always revert any literal override to the design-time default value by 
selecting the “Use default value from package” option in the Set Parameter 
Value dialog, as shown in Figure 5-7. 
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Ua Set Parameter Value = o x 


Parameter: 





Data Source =(local)\Demo: Initial A 
© Edt value: Catalog=Test DB; Integrated 





Figure 5-7. Reselecting the design-time default 


The value text decoration returns to none and the value reverts to the 
design-time default, as shown in Figure 5-8. 
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Value 
ConnectionSinng Data Source svmDemo\Demo:Intial Catalogs TestDG:ht.. L] 


\ndiaiCatalog ‘Testo l 
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yf mieno Deno [PHIEN A. 
Ray Leonard] 


Serverhiares wmDemo' Demo 
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Progress 
Hesg 


Figure 5-8. Reverted to design-time default 


Externalizing the Connection Configuration 


So far you’ve examined two sources of SSIS catalog project configurations 
settings: design-time defaults and literal overrides. You probably noticed 
a third option on the Set Parameter Values dialog, one that was disabled. 
“Use environment variable” is the third option but it requires additional 
configuration before this option is enabled. 

Before we jump into the next demo, let’s discuss how the SSIS 
catalog manages externalization. Externalization is storing values in 
some other location for use at runtime. The SSIS catalog’s mechanism 
for externalization is SSIS catalog environments. Catalog environments 
contain a collection of catalog environment variables which, in turn, 
contain values that are used at runtime to override catalog project 
parameter values. 

That last paragraph is relatively short for the depth and importance of 
the information contained therein. Please let it sink in before proceeding. 

Let’s talk about the SSIS catalog’s externalization mechanism in more 
detail. Let’s begin with an SSIS catalog environment, as shown in Figure 5-9. 
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Catalog 
Environmem 





Figure 5-9. A catalog environment (figuratively) 


The SSIS catalog environment contains a collection of zero or more 


SSIS catalog environment variables, as shown in Figure 5-10. 










Catalog 
Ervironmern 


Figure 5-10. A catalog environment variable in a catalog 
environment 


To connect an SSIS catalog environment to an SSIS catalog project, 
the SSIS catalog uses a reference. A reference is simply a relationship that 


connects a catalog project to a catalog environment, as shown in Figure 5-11. 


LF Treterenc > 





Figure 5-11. A reference relating an SSIS catalog project to an SSIS 
catalog environment 
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In order for a parameter in a catalog project to consume a catalog 
environment variable value, a reference mapping is created. A reference 
mapping maps a value stored in an SSIS catalog environment variable to a 
parameter in an SSIS catalog project, as shown in Figure 5-12. 


Reference 
Mapping 





Figure 5-12. A reference mapping between an SSIS catalog project 
parameter and an SSIS catalog environment variable 


To restate, because this can be a challenging topic, you first create 
and configure an SSIS catalog environment. Configuring the catalog 
environment includes defining SSIS catalog environment variables. That’s 
the first step in externalization. 

You next create a reference between an SSIS catalog environment 
and an SSIS catalog project. Creating a reference is the second step in 
externalization. 

You may then assign SSIS catalog environment variables to SSIS 
catalog project parameters via the reference. This is reference mapping. 
Mapping a parameter value to a catalog environment variable value via a 
reference is the third step in externalization. 


Let’s walk through this procedure in a demo. 


Creating an Environment 


In the SSIS catalog, right-click the Environments virtual folder found in 
the Test catalog folder and then click Create Environment, as shown in 
Figure 5-13. 


71 


CHAPTER 5 CONFIGURE THE SSIS CATALOG PROJECT 


E © Integration Services Catalogs 
= F ssispB 
© Framework 
=) Ge Test 
=) eg Projects 
& AÀ DILMSample 
D a Environments 


© 3 SOL Server Agent 






Create Environment... 
Start PowerShell 
Reports b 


Refresh 
Figure 5-13. Preparing to create a catalog environment 


When the Create Environment dialog displays, provide an 


environment name and optional description, as shown in Figure 5-14. 


Be Create Environment zg o x 
(@ Ready 
Select a page ET Script ~ @) Help 
# General 
Environment name: 
lenvConnection 1 | 
Environment description: 
Connection 


yf vmDemo\Demo [VMDEMO‘A 
Ray Leonard] 





E] rr z | Cancel | | Help 
Figure 5-14. Creating an SSIS catalog environment 
Click OK to close the Create Environment dialog. 
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Configuring an Environment 


Expand the Environments virtual folder in the SSIS catalog. Right-click the 


new catalog environment and click Properties, as shown in Figure 5-15. 


=| © Integration Services Catalogs 
= [À SSISDB 
+ © Framework 
=) @& Test 
=] Æ Projects 
# AÀ DILMSample 
= © Environments 





i enve onnection] | 


I yo SAL Server Agent Move... 
start PowerShell 
Reports + 


Rename 


Delete 
Refresh 
__ Properties 


Figure 5-15. Preparing to configure the SSIS catalog environment 
variables 


Click the Variables page. Add a name for your catalog environment 
variable (I named mine ConnectionString). Set the type to String. Add a 
valid connection string to the Value field, as shown in Figure 5-16. 
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Be Environment Properties = o x 
Select a page E Script ~ @) Help 
| # General 
E Variables 
A Permissions vests 
Name | Type Dhemeription alu Sensi 
Connection Sring | Sting w | Data Sources, Demo Initial Catalog = Test0GIntegated Secuiys Taus Äppicsiiy Oo 
| ee | O 
Connection 


Rendy 
Remove 


Cancel || Help 
Figure 5-16. Configuring an SSIS catalog environment variable 


Click the OK button to close the Environment Properties dialog. 


Configuring a Reference 


Right-click the DILMSample catalog project and click Configure. When the 
Configure - DILMSample dialog opens, click the References page. Click the 
Add button and select the catalog environment you just created beneath 
the Local Folder (Test) virtual folder in the Browse Environments 
dialog, as shown in Figure 5-17. 
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ii References: 
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la Browse Environments x 

Connection 
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Ray Leonard] 





[ok | | cance a 


Figure 5-17. Selecting the catalog environment for the reference 


There are actually two “paths” to the envConnection1 catalog 
environment available in the Browse Environments dialog. The Local 
Folder path creates a “relative” reference in SSIS catalog parlance. 

If you expand the SSISDB > Test folder, you could select the same 
envConnection1 catalog environment. This would create an “absolute” 


reference. 


Click the OK button to select the catalog environment for the reference. 


The Configure - DILMSample references page will appear similar to that 


shown in Figure 5-18. 
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Se Configure - DILMSample 

\@ Ready 

Select a page LJ Script ~ @) Help 
# Parameters 
# References plantas 

Environment Environment Folder 






envionnection 1 


yi vmDemo'\Demo [VMDEMO\A. 
Ray Leonard] 





Figure 5-18. Reference configured! 


Please note the “.” in the Environment Folder column; it indicates you 
selected a local catalog environment, or a catalog environment from the 


same catalog folder as the catalog project. 


Configuring a Reference Mapping 


Click the Parameters page and then click the Connection Managers tab. 
Click the ellipsis beside the ConnectionString property for your lone 
connection manager to open the Set Parameter Value dialog. Please note 
all three value configuration options are now enabled. Select the “Use 
environment variable” option and then select the ConnectionString 
catalog environment variable from the dropdown, as shown in Figure 5-19. 
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‘da Set Parameter Value =- o x 


Parameter: 








Value 
C) Edit value: 
C) Use default value from package [Data Source=vmDemo\Demo: Initial Catalog=1 


(©) Use environment variable: 


Lonnection String 





Figure 5-19. Selecting the catalog environment variable for the 
reference mapping 


Click the OK button to create the reference mapping between the SSIS 
catalog project parameter, SimplePackage.dtsx’s vmDemo\Demo\testDB1 
connection manager ConnectionString, and the SSIS catalog environment 
variable named ConnectionString found in the reference to the SSIS catalog 
environment you created a few minutes ago. Your Configure - DILMSample 


dialog should now appear similar to that shown in Figure 5-20. 
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Parameters Connection Managers 


| Container Name Properties 
Se ooe o 
[Comectionting  |ConnectionSima 5 
| itiaiCatalog Te08 
| Password 
| Retain SameConnection | Fale 
| ServerName vmDemo\Demo 
“UserName NULL 


Figure 5-20. Completed reference mapping! 


Please note that the value of the ConnectionString property is now the 
name of the SSIS catalog environment variable, and the text is decorated 
with an underline. 


Testing the Configuration 


To test your configuration, you will execute the SSIS package in the SSIS 
catalog. Right-click SimplePackage.dtsx and click Execute, as shown in 
Figure 5-21. 


=] © Integration Services Catalogs 
=) [Ay SSISDB 
a E Framework 
o me Test 
=) © Projects 
=] AÅ DILMSample 
=) E Packages 
Gi simplePackage.dtsx 
fH © Environments Configure... 
a) Æ SOL Server Agent | Execute... 


Validate... 








Figure 5-21. Executing SimplePackage.dtsx 
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You'll see an error message at the top of the Execute Package dialog 
when the Execute Package dialog displays. Click the error to see the error 


message displayed in a message box, as shown in Figure 5-22. 


Bie Execute Package - SimplePackage.dtsx(DILMSample -— E 

The property "ConnectionString" of connection manager ym Dema Demo Tet DET" is configured to use an environment variable, but no environment has been selected. Select the.. 
‘Select a page IT Script ~ @ Hep 

# General 


Parameters Connection Managers Advanced 


Parameter Value 

ireProjParen Ja Laj 
a ‘StingProjParam [Apod parameter EA 
| | rtPkgParam a2 Was 
|| StingPkgParam [Hi There! 





Microsoft SQL Server Management Studio x 


Ta The peoperty “Commection String” of connection manager "vmDemo\Demo. TestDE1" cd ca ec 
G ti age an environment variable. but no environment has been selected. Select the “Environment” checkbox and 
: ~ speck the environmen to use, or specty a literal valve for the property, 
yF mieno Deno [VMDEMOVA. 
| 1 L 


i 


Ready 
C Environment: Mam eored] 


OK Cancel Help 
Figure 5-22. Displaying the execute package error message 


To clear the error, follow the instructions included in the error 
message. Check the Environment checkbox and select an SSIS catalog 


environment from the dropdown, as shown in Figure 5-23. 


79 


CHAPTER 5 CONFIGURE THE SSIS CATALOG PROJECT 
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Figure 5-23. Select an SSIS catalog environment 


The execute package error clears. Click the OK button to execute the 
SimplePackage.dtsx SSIS package. A message box similar to that shown in 
Figure 5-24 informs you that package execution has started and asks if you 


would like to view the Overview Report. 


Microsoft SQL Server Management Studio 


% The operation, ID 10068, has started. For information about package executions and validations, open 
| the Integration Services Dashboard report from the SSISDB node. For information about other 
Integration Services operations, open the All Operations report from the SSISDB node. 





Would you like to open Overview Report now? 





ay No 





Figure 5-24. Overview Report prompt 


Click the Yes button to display the Overview Report, shown in 
Figure 5-25. 
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Figure 5-25. The Overview Report 


Please note the value for the vmDemo\Demo . TestDB1.ConnectionString 
parameter. The value at runtime is the value of the ConnectionString 
catalog environment variable you configured in the catalog environment 
named envConnection1. This test verifies that the reference mapping is 
configured and working properly. 
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Conclusion 


These are the three sources of values for SSIS parameters in the SSIS 
catalog (with matching text decoration): 


1. Design-time defaults 
2. Literal overrides 


3. Reference mappings 


Design-time defaults remain stored in the SSIS catalog but may be 
overridden using literal overrides or references and reference mappings to 


catalog environments and environment variables, respectively. 
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Catalog Browser 


I was honored to be a Microsoft SQL Server MVP for five years (2007-2012). 
One cool thing about being a Microsoft MVP was access to the internal 
developer teams. Everyone could file Microsoft Connect items to report 
bugs and make suggestions for product improvements. Many MVPs did so 
only to have their bug reports marked as “works as designed” or “won't fix” 
and suggestions responded to with something similar. It was discouraging. 
There are reasons many Connect items were addressed in this way. Iam 
happy to report the root cause (performance-based management, or PBM) 
has been abandoned and the Microsoft Developer Teams are really and 


truly listening and responding to requests from the field. 


Why I Built DILM Suite, by Andy Leonard 


That doesn’t mean every suggestion is acted upon (I promise this is 

not a complaint). It turns out that Microsoft is a software development 
enterprise. As big as Microsoft is, it can’t possibly respond to every request. 
When I realized this, I began thinking about how I might address gaps 

I perceived. I’d cofounded a consulting company and we (collectively) 
weren't interested in becoming a software product company. But I was 
very interested in developing products to address gaps in data integration 


lifecycle management (DILM). 
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In 2015 I left the consulting company I cofounded and immediately 
began developing the software I’d dreamed of building. In my opinion, the 
most fair answers to the question of “Why?” are the following: 


1. Icame to believe the Microsoft SSIS Developer 
Team would never address the things I perceived as 
“gaps” in the product story. 


2. Icame to believe that the consulting company I 
cofounded and I held irreconcilable visions of how 
to address DILM issues. 


Looking back with two years of perspective, I believe focusing on 
DILM was the best long-term move for me. I started another consulting 
company, Enterprise Data & Analytics (entdna.com), mostly to fund my 
coding habit. 


Surfacing the SSIS Catalog 


Let’s examine the SSIS catalog surface in the SSMS Object Explorer’s 
Integration Services Catalogs node, shown in Figure 6-1. 


=) & Integration Services Catalogs 
3 [A SSISDB 
4) © Framework 
=) E Test 
=) E Projects 
- ish DILMSample 
= W Packages 
11 SimplePackage.dtsx 
=) © Environments 
O envConnection| 


Figure 6-1. The SSIS catalog as shown in the SSMS Object Explorer 
Integration Services Catalogs node 
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Beneath the Integration Services Catalogs node we find the SSIS 


catalog named SSISDB. Two catalog folders are displayed, Framework and 


Test. The Test folder contains the Projects and Environments virtual 
folders. The Projects virtual folder contains the SSIS catalog project 
named DILMSample, which in turn contains the SSIS package named 


SimplePackage.dtsx. The Environments virtual folder contains the catalog 


environment named envConnectionl. 


You know, because you've done the work, that there’s more there than 


meets the eye. 


SSIS Catalog Environment Configuration 


If you double-click envConnection1, you can see details of your catalog 
environment variable on the Variables page, shown in Figure 6-2. 


Ar Environment Properties - o x 

O Foxy 

Select a page OJ Script + @ Help 

E Genen 

# ‘Vorsbles 

F Peri wemasbles: 
Name Type Despngtoni Valus Serene 
Laprireectace Share| | Stang ki Dieta Source- Demo hia Cataieg= Tat 6 integrated Secusty= Teuactpeheation Name=5, T 

i J | = 


Figure 6-2. Viewing the Variables page of an SSIS catalog 
environment 


The Variables page contains details about SSIS catalog environment 
variables including name, data type, description, value, and whether the 


variable is sensitive. 


SSIS Catalog Project Configuration 


The Parameters tab on the Parameters page of the SSIS Catalog Project 
Configuration dialog lists SSIS project and package parameters, their 


container name, and value by default, as shown in Figure 6-3. 
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Figure 6-3. Viewing project parameters and values for an SSIS 
catalog project 


The Connection Managers tab of the Parameters page contains a list 


of SSIS project and package connection managers and their properties, as 
shown in Figure 6-4. 


Be Configure - DILMSample 






































@ Ready a i 
Select a page “DT Script -| @ Help 
# Parameters 
# References : 
Parameters Connection Managers 
SEES 
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Figure 6-4. Viewing connection manager parameters and values for 
an SSIS catalog project 
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The References page of the SSIS Catalog Project Configure dialog 
contains a list of SSIS catalog environments the SSIS catalog project may 
reference at runtime, as shown in Figure 6-5. 


we Configure - DILMSample 
O Ready 
Select a page {J Script ~ @ Help 


J Parameters 
# References 


References: 


Environment Folder 





Figure 6-5. Viewing project references for an SSIS catalog project 


That’s a lot of right- and double-clicking just to see what’s configured 
in an SSIS catalog project. 


Catalog Browser 


The SSIS catalog is filled with really cool and useful configuration 
information, but one has to know where to look and, in some cases, where 
to look isn’t so obvious. 

Enter Catalog Browser, a free utility that is part of the DILM Suite and 
available at dilmsuite.com/catalog-browser. Catalog Browser was built 
to surface the contents of the SSIS catalog in a single view: a tree that 
exposes all relevant SSIS catalog artifacts, properties, and configurations. 

As shown in Figure 6-6, Catalog Browser surfaces the same metadata 
as the SSMS Object Explorer Integration Services Catalogs node. 
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Æ SSIS Catalog Browser v0.4.1.0 (beta) 
View Help 
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[~] Use Integrated Security 
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6- envConnection1 


Figure 6-6. Catalog Browser surfacing part of the SSIS project and 
configurations metadata 


Looking at Figure 6-6, though, you probably already see some 
differences between Catalog Browser and the SSMS Object Explorer 
Integration Services Catalogs node. Note the Project Parameters and 
Project References virtual folders present beneath the SSIS catalog 
project, in addition to the Packages virtual folder. 

Expanding these virtual folders reveals the SSIS catalog project 


parameters and reference, as shown in Figure 6-7. 
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Æ SSIS Catalog Browser v0.4.1.0 (beta) 
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Figure 6-7. SSIS catalog project parameters and references 


Remember in Figure 6-3 the SSMS Object Explorer Integration Services 
Catalogs node surfaced all parameters: SSIS catalog project parameters 
and SSIS package parameters. Where are the package parameters? They’re 
here in Catalog Browser. To view the package parameters, expand the 
SimplePackage.dtsx SSIS package node, as shown in Figure 6-8. 
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Figure 6-8. Viewing SSIS package parameters 






Recall that connection manager properties are treated as parameters 
in the SSIS catalog. They are prefixed with “CM. You can see that the 


SSIS package connection manager vmDemo\Demo. TestDB1 connection 


string property is mapped to an SSIS catalog environment variable named 


ConnectionString. 


To surface the reference used for the reference mapping, expand the 


Package References virtual folder, as shown in Figure 6-9. 
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Figure 6-9. Viewing the package reference 


Expanding the Package Reference virtual folder surfaces the Test/ 
envConection1 catalog environment. Expanding the Test/envConection1 
catalog environment reveals that the catalog environment variable named 
ConnectionString is mapped to the vmDemo\Demo. TestDB1 connection 
string property. 

But what’s the value of the ConnectionString catalog environment 
variable? Expand the envConnection1 catalog environment in the 
Environments virtual folder to view the collection of catalog environment 
variables, their data types, and their values, as shown in Figure 6-10. 
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Figure 6-10. Catalog environment variables, data types, and values 


SSIS package properties includes a Package Version property 
constructed from the Version Major, Version Minor, and Version Build 
properties of the SSIS package. Every time a developer saves an SSIS 
package, the Version Build property increments. It’s possible to revise an 
SSIS package and “trick” the Version Build property by manually setting it. 
I have not yet found a valid use case for doing so to SSIS catalog-deployed 
SSIS packages. 

The Package Version property can be used to detect different versions 
of SSIS packages deployed to an SSIS catalog. Because SSIS developers can 
manually set the Version Build property, Package Version is not a reliable 
indication. 

The Package Properties virtual folder surfaces SSIS package 
metadata, as shown in Figure 6-11. 
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Æ SSIS Catalog Browser v0.4.1.0 (beta) 
View Help 


Catalog 1 


SQL Server Instance: |vmDemo\Demo 
M] Use Integrated Security 














2: SSISDB 
H- Catalog Properties 
H- Framework 
D-H Test 
=) Projects 
S-a DILMSample 
=}- Packages 
S- T) SimplePackage.dtsx 
S- Package Properties 
= Package Version: 1.0.5 
-> Package Version Comments: 
-> Package GUID: 1c6c5c53-7317-40b3-862144d56e58dd97 
Œ- Package Parameters 
Œ- Package References 
-m Project Parameters 
H- Project References 
H- Environments 


Figure 6-11. SSIS package properties 


Catalog properties are handy for detecting differences in patch levels 
(via the Schema Build property). Catalog Version is a property exposed by 
Catalog Base, the custom catalog object that lies beneath Catalog Browser. 

Catalog Base works with SSIS 2012, 2014, and 2016 catalogs. 


The Catalog Properties virtual folder surfaces SSIS catalog metadata, 
as shown in Figure 6-12. 
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View Help 


Catalog 1 
SQL Server Instance: | vmDemo\Demo 


V] Use Integrated Security 





B- T SSISDB 
F- © Catalog Properties 


i von > Sy ees ee 0 

| ven > Server Logging Level: 1 

-> Server Operation Encryption Level: 2 
7 J Version Cleanup Enabled: True 
H- Framework 
H-D Test 


Figure 6-12. SSIS catalog properties 


Conclusion 


Catalog Browser surfaces SSIS catalog artifacts, configurations metadata, 
and artifact properties in a single view. 
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SSIS Catalog 
Compare 


Viewing the SSIS catalog contents, including SSIS catalog projects, 
packages, environments, environment variables, references, and reference 
mappings along with SSIS package and SSIS catalog properties, is helpful. 
But what if you want to see the differences between the contents of one 
SSIS catalog instance and another? Or, perhaps even more useful, how can 
you know the enterprise QA and Production SSIS catalog instances match? 


Why I Built SSIS Catalog Compare, by Andy 
Leonard 


My team built a fairly complex data integration solution for a client using 
SSIS. We tested the solution in two DevOps tiers, Test and UAT (User 
Acceptance Testing). We identified some issues and corrected them. It was 
then time to deploy to Production. My team and I were on standby during 
the Production deployment and initial Production tests. 

Initially, everything failed. 

We identified the root causes and corrected the issues, but we had egg 
on our collective faces with the business, and for good reason. We had 
assured them “we were doing it right.” We were, with one exception. The 
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enterprise data architect identified the gap with one very good question, 
“How can we know the Production and UAT SSIS catalogs are the same?” 
My response at the time, “[Pregnant pause while thinking... then] I don’t know.’ 

I built SSIS Catalog Compare so I—and my customers, and you—can 
answer that question. 


SSIS Catalog Compare 


You can begin by thinking of SSIS Catalog Compare as two Catalog 
Browsers. Like Catalog Browser, SSIS Catalog Compare uses Catalog Base, 
a custom catalog object. In SSIS Catalog Compare, Catalog Base is used to 
populate two trees, each representing a different SSIS catalog, as shown in 
Figure 7-1. 





AE SSIS Catalog Compare v2.0.2.0 = E ~*~ 
View Help 
Catalog 1 a = = Catalog 2 —————— 
SQL Server Instance: [vm Demo Dema Clear SQL Server Instance: |vmDemo'\Dew Refresh Gear 
| S-o vmDemo\Demo || a-o vmDemo\Dev 
S-ip SSISDB -ip 551508 

H- Catalog Properties Gp Catalog Properties 

H Framework H- Framework 

HH Test 

SSIS Catalog vmDemo\Demo loaded 
Compare 


Figure 7-1. SSIS Catalog Compare 


Once two catalogs are loaded, they may be compared by clicking the 
Compare button. As shown in Figure 7-2, SSIS Catalog Compare uses the 
italics font to indicate that a difference has been detected beneath a node 
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and a different background color to indicate artifacts that are present in 
one SSIS catalog but missing from the other. 


AE SSIS Catalog Compare v2.0.2.0 = E ~*~ 


View Help 


Catalog 1 aee : 
SQL Server Instance: |ymDemo\Dema Refresh Clear 





| Refresh Clear 


Comparison of vmDemo\Demo and vmDemo\Dev completed, 


Figure 7-2. After a Compare operation 


The Test catalog folder in the VmDemo\Demo catalog does not exist in 
the vmDemo\Dev catalog, hence the background color on the vmDemo\Demo\ 
SSISDB\Test node. The “deeper differences” indications (italics fonts) 
shown in Figure 7-2 indicate differences within the Framework catalog 
folders and between the catalog properties. 


Expanding the Differences 


One time-saver is Expand Differences, shown in Figure 7-3. 
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Æ SSIS Catalog Compare v2.0.2.0 


View Help 
Catalog 1 






























Deploy Folder... 
Deploy Folder and Contents... 
Deploy Folder Differences... 


Generate Folder Script... 
Generate Scripts for Folder and Contents... 


Generate Scripts for Differences... 


Delete Folder... 
Delete Folder and Contents... 
Export Folder... 


Expand All... 


| Expand Differences... 


Collapse All... 








Figure 7-3. Expand Differences 


Expand Differences will expand nodes above differences and the nodes 
that are different. In large SSIS catalog projects, catalog projects with lots 
of configurations metadata, or both, Expand Differences can reduce the 
number of nodes expanded. Expanding the differences in the Framework 
catalog folder for both catalogs, you see the Parent.dtsx SSIS package 
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SSISDB connection manager’s ConnectionString property is configured for 
each server's local SQL Server instance, as shown in Figure 7-4. 


Æ SSIS Catalog Compare v2.0.2.0 - o x 
View Help 
1 Catalog 2 
SQL Server instance: wrDemo Deno Retresh dene SQL Server instance: vmOemo\Dev Retest Gear 
» Properties Popetes 
or 3 
a vcs 
Geri Met act al osder GeriMetadst s Loader 
Kare nox 
Packages Packages 
za) aa i) Aeren 
E O Package Propetes E O Package Propetes 
Package Pameten 


we ioicaonlisme [Strng) Framework Tex 


ww Source -vmeDemo\Demo in a 
& CM SSISOB. ntaCataiog [Stng} SSISO8 we? CM SSISOE nasiCatsiog (Sng) SSIS08 


Figure 7-4. Different ConnectionString property values 


This is a really good thing because, if these values matched, framework 
application executions in one catalog would start SSIS executions in 


another catalog. That could be bad. 


Catalog Properties 


I didn’t discuss catalog properties in the section about Catalog Browser 
because I wanted to cover catalog properties here in more detail. If you 
collapse the Framework catalog folder and expand Catalog Properties, you 
see the difference is the Schema Build property, shown in Figure 7-5. 
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AL $515 Catalog Compare v2.0.2.0 = (m) k 
View Help 
Catalog 1 Catalog 2 
SUL Server Instance: | vmDemo\Demo Refresh dear SOL Server Instance: vm Demo Dew Refresh dear | 
bihotzian kimanin AA | 
vmbemo Der | 
E £ E 
c 0 EME 
> Encryption Algorithm: AES_256 T tery r Ever Apoi AES_256 
= Max Project Versions: 10 j = Max Project Versions: 10 
= Operation Geanup Enabled: True i -= Operation Geanup Enabled: True 


=> Retention Window: 365 | > baton aoe i 
-> Schema Buld: 13.0.1601.5 | = Schema 
+ Catalog Version: 2016 > Catalog orrn 2016 
— Schema Version: 4 io be > Schema Version: 4 
> Server Customized Logging Level: 0 | = Server Customized Logging Level: 0 
—=} Server Logging Level: 1 / h Server Logging Level: 1 
= Server Operation Encryption Level: 2 > Server Operation Encryption Level: 2 
= Version Geanup Enabled: True > Version Qeanup Enabled: True 
Ge) Framework E bi Framework 



































Comparison of vmDemo'\Demo and vmDemo'\Dev completed. 


Figure 7-5. Different values for the SSIS catalog Schema Build 
property 


Note from the property just beneath Schema Build that both of these 
SSIS catalogs are SSIS 2016. vmDemo\Demo’s Schema Build version tells us 
that this catalog is the General Availability (or GA, formerly known as RTM 
[Release to Manufacturing]) version of SSIS 2016. vmDemo\Dev’s catalog is 
an SSIS 2016 SP1 Schema Build version. 

The Catalog Base object is built to allow SSIS Catalog Compare to 
compare SSIS catalogs from different releases and versions of SQL Server. 
For example, I could compare the contents of an SSIS 2012 catalog to those 
of an SSIS 2016 catalog. SSIS 2016 added a couple properties to the catalog 
properties, so you would see a couple properties marked on the 2016 side 
of the compare as missing from the 2012 side. But the Compare operation 
would succeed and produce accurate and useful results. 
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SSIS Catalog Compare Scripting 


In many enterprises with two or more IT professionals, the professional 
who develops the software is not permitted to deploy that software. 
Large enterprises have entire teams dedicated to release management. 
Lifecycle management will help enterprises of any size reduce downtime 
and improve code supportability and maintainability. This holds for SSIS 
because data integration lifecycle management is just as vital as web, GUI, 
and middle-tier software lifecycle management. 

SSIS Catalog Compare is designed to support DevOps and enterprise 
DILM. 

You can script individual artifacts using SSIS Catalog Compare, but 
perhaps the more useful (and quicker) functionality is to script a catalog 
folder and all its contents, as shown in Figure 7-6. 


Æ SSIS Catalog Compare v2.0.2.0 


View Help 
Catalog 1 


SQL Server Instance: Refresh Clear 


|e o vmDemo \Demo 
| D-M ssisoe 
+) | Framework 
Hi] 
| Deploy Folder... 











Deploy Folder and Contents... 
Deploy Folder Differences... 


Generate Folder Script... 
Generate Scripts for Folder and Contents... 


Generate Scripts for Differences... 





Delete Folder... 











Figure 7-6. Preparing to generate scripts for a catalog folder and its 
contents 


101 


CHAPTER 7 SSIS CATALOG COMPARE 


When you click “Generate Scripts for Folder and Contents,’ SSIS 
Catalog Compare prompts you for a file system folder in which to store the 
scripts, as show in Figure 7-7. 


Browse For Folder X. 





> B A. Ray Leonard A 
v E This PC 
> E Desktop 
v |=) Documents 
> F! BuildFromScratch 
| DILM 


Figure 7-7. Selecting a target file system folder for the scripts 


When you select a file system folder, SSIS Catalog Compare generates 
the scripts and ISPAC files to create the catalog folder and all its contents, 
as shown in Figure 7-8. 


Th) ok - 
| Fie | Home Share View 
e E {4 » ThisPC » Documents > DILM > vmDemo_ Demo > Test v & 
v | DILIM mt 
v 1 | vmDemo_Demo J 1_vmDemo-Demo_SSISOB_Test. folder.sq! 
Test EÈ 2.vmDemo-Demo_SSISDB_DILMSample.ispac 

Be JB 4_vmDemo-Demo_SSISOB_Test_envConnection !.environment.sq! 

_ . SimpleLoader JD S_vmDemo-Demo_SSISOB_Test_DILMSample_envConnection! .reference.sql 

a Snagit ə 5_vmDemo-Demo_SSISOB_Test_DiLMSample_SimplePackage_envConnection 1 reference.sq! 


Figure 7-8. Viewing the target file system folder for the scripts 


The scripts are numbered in an order that represents dependencies. 
For example, you cannot deploy an SSIS project using the ISPAC file, 
number 2, unless and until the SSIS catalog folder for that SSIS catalog 
project exists, script number 1. 
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Double-clicking the files in the order listed will create the Test catalog 
folder (script 1) on the target SSIS catalog, deploy the DILMSample SSIS 
project using the Integration Services Deployment Wizard GUI (you 
will need to select the Test catalog folder as the target catalog folder 
during deployment), create the envConnection1 catalog environment 
and its catalog environment variable(s), create a reference between the 
DILMSample SSIS project and envConnection1, and create another 
reference with reference mapping(s) between the SimplePackage. dtsx 
SSIS package and the envConnection1 catalog environment. 

Why two reference files? Inside the SSIS catalog references for SSIS 
catalog projects and SSIS packages are distinct artifacts. 


Creating a Catalog Folder 


The scripts (and ISPAC) are idempotent, or re-executable, as shown in 
Figure 7-9. 





1 

2 | Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo_Demo\Test \\ 
3 Generated From Catalog Instance: vmDemo\Demo 

4 \Catalog Name: SSISDB 

5 | Folder Name: Test 

6 | Generated By: VMDEMO\A. Ray Leonard 

7 | Generated Date: 8/8/2017 10:65:38 PM 

8 | Generated From: CatalogBase v2.0.2.0 executing on: VMDEMO 
g | 
18 
11 Use SSISDB 
12 go 
13 


14 =print ‘Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo_Demo 
15 | Generated From Catalog Instance: vmDemo\Demo 

16 | Catalog Name: SSISDB 

17 | Folder Name: Test 

18 | Generated By: VMDEMO\A. Ray Leonard 

19 Generated Date: 8/8/2017 10:05:38 PM 

20 | Generated From: CatalogBase v2.0.2.0 executing on: VMDEMO' 


Figure 7-9. Idempotent Transact-SQL scripts 
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If the Test catalog folder does not exist, the script creates it. If the Test 
catalog folder exists, the script outputs a message informing the executor 
of this fact. 

The output messages are written to be copied and pasted into a ticket’s 
Notes field before the ticket is closed. The messages provide detailed 
information about how and when the script was generated, by whom, 
when and where the script was executed, and by whom, as shown in 
Figure 7-10. 


El Messages 
Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo_Demo\Test\\ 
Generated From Catalog Instance: vmDemo\Demo 
Catalog Name: SSISDB 
Folder Name: Test 
Generated By: VMDEMO\A. Ray Leonard 
Generated Date: 8/8/2017 10:05:38 PM 
Generated From: CatalogBase v2.0.2.0 executing on: VMDEMO 


Folder SSISDB\Test 
- Creating Test folder 
- Test folder created 


Figure 7-10. Script output messages 


Refreshing the SSIS catalog node in SSMS Object Explorer will reveal 
the Test catalog folder has been created, as show in Figure 7-11. 
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| © S Fra Active Operations 
Create Folder... 
Customized Logginc 


Start PowerShell 
Database Upgrade 
Execute in Scale Out. 


Manage Scale Out... 


D 


Properties 
Figure 7-11. Refreshing the SSIS catalog node 


Figure 7-12 shows the results of the refresh: the Test catalog folder has 
been created by the script. 


= ™ Integration Services Catalogs 
5 [A SSISDB 
%) © Framework 


T: 


Figure 7-12. The Test catalog folder lives! 
I began this section stating, “The scripts (and ISPAC) are idempotent...” 
What happens if you re-execute the script you just used to create the Test 


catalog folder? Let’s return to SSMS and try it. The results are shown in 
Figure 7-13. 
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EH Messages 
Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo Demo\Test\\ 
Generated From Catalog Instance: vmDemo\Demo 
Catalog Name: SSISDB 
Folder Name: Test 
Generated By: VMDEMO\A. Ray Leonard 
Generated Date: 8/8/2017 10:05:38 PM 
Generated From: CatalogBase v2.0.2.0 executing on: VMDEMO 


Deployed to Instance: VMDEMO\DEV 
Deploy Date: 08/09/2017 06:27:00 


Folder SSISDB\Test 
- Test folder already exists. 
- Setting Test folder description to "" 
- Test folder description set to "™" 


Figure 7-13. Re-executing the Catalog Folder script 


Please note the message returned: “Test folder already exists.’ The Test 
catalog folder was not harmed by re-executing the script. The script simply 
checks for the existence of the catalog folder and creates it if it does not 
exist. If the catalog folder exists, the script returns a message. The Transact- 
SQL that performs this operation is shown in Figure 7-14. 
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print ‘Folder SSISDB\Test' 
If Not Exists(Select * 
From SSISDB.[catalog].folders 
Where name = N‘Test' ) 
begin 
print ' - Creating Test folder ' 
declare @folder_id bigint 
Exec SSISDB.[catalog].create_folder 
@folder_name = N'Test' 
, @folder_id = @folder_id OUTPUT 
print ' - Test folder created’ 
end 
else 
begin 
print ' - Test folder already exists.' 
end 


Figure 7-14. Transact-SQL that first checks for catalog folder 
existence 


If you refresh SSIS catalog Compare Catalog 2, you now see the Test 
catalog folder, as shown in Figure 7-15. 


Catalog 2 
SQL Server Instance: 








Figure 7-15. The Test catalog folder created 
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Deploying the SSIS Project 


To deploy the SSIS project, double-click (or right-click and click Open) the 
ISPAC file, as shown in Figure 7-16. 


a DILM A Name 
L vmDemo_Demo JD 1_vmDemo-Demo_SSISDB_Test.folder.sql 
Test pà, ZvmDemo-Demo_SSISOB_DILMSample.ispac 
[a R JB 4 vmDemo-Demo_SSISDB_Test_envConnec’ Open 


Figure 7-16. Opening the ISPAC file 


The Integration Services Deployment Wizard starts; this is the same 
wizard used to deploy SSIS projects from SQL Server Data Tools (SSDT). 
Stepping through the wizard, the first stop is the Select Source page shown 
in Figure 7-17. 


A} Integration Services Deployment Wizard = E k 


* Select Source 
Ez 4 


introduction ik) Help 
Select Destination Selaci the Seniors moe Beant Pek ¥ 
Review Select the Integration Services project that you want to deploy. 

Resuts 


(@) Project deployment file 
C) Integration Services catalog 


Path: 





C’\Users\A. Ray Leonard’\Documents\DILM\vwmDemo_Demo\\Test'2_v} Browse... 


Figure 7-17. The Integration Services Deployment Wizard Select 
Source page 


This page surfaces a lot of functionality. The Deployment Model 
dropdown is used to select project or package deployment. Beginning with 
SSIS 2016, operators have the option of deploying the entire SSIS project, 
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which was the only option available in SSIS 2012 and 2014, or deploying 
a single SSIS package. Operators may also select the type of SSIS project 
source. There are two options available: a project deployment file or an 
Integration Services catalog. Since you started this exercise by opening an 
ISPAC file, the project deployment file option is selected for you. 

If you select the Integration Services catalog option, the Select Source 
page presents catalog project settings that may be configured for an SSIS 
catalog project deployed to a different SSIS catalog, as shown in Figure 7-18. 


ar 


Integration Services Deployment Wizard = m x 
F 
Pie Select Source 





Select the deployment model: Project Deployment we 


Select Destination 


Review Select the Integration Services project that you want to deploy. 


Results 


CI Project deployment file 
(©) Integration Services catalog 


Server name: 
Path: 
/SSISOB/Test/DILMSample | 


Figure 7-18. Deploying from one SSIS catalog to another 


The remainder of the Integration Services Deployment Wizard pages 
and process are the same as those covered in Chapter 4. On the Select 
Destination page, enter the name of the SQL Server instance that hosts 
the target SSIS catalog. You can then browse to the newly-created catalog 
folder, shown in Figure 7-19. 
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PEA a es 
AÀ Select Destination 
Enter the destination server name and where the project will be located in the 
EXT eoon Seca caniao. 
Foa Server name: 
Results | vmDemo\Dev | Browse 
Path: 
| 
Browse for Folder or Project o x 
Select a folder for the project or replace an existing project in an 
Integration Services catalog 
| © i SSISDB 
H-a Framework 
oe 





OK Cancel 


Figure 7-19. Browsing to the Test catalog folder recently created 


Step through the remainder of the Integration Services Deployment 
Wizard until the SSIS project has been deployed, as shown in Figure 7-20. 
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af Integration Services Deployment Wizard 


ry) 
An Results 


D Loadng project 
@ Connecting to destination server 


SSIS CATALOG COMPARE 











Figure 7-20. A successful deployment 


Refresh SSIS Catalog Compare to see that the SSIS catalog project now 


exists, as shown in Figure 7-21. 
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Catalog 2 
SQL Server Instance: | vmDemo\Dev | Refresh Clear 


= 
=)-\p SSISDB 


H- Catalog Properties 
+) Framework 
&)- 9) Test 
=). | Projects 
=}--$s, DILMSample 
9-1 Packages 
+- 14) SimplePackage dtsx 
+) Project Parameters 


Figure 7-21. Viewing the DILMSample SSIS catalog project 


As with the Catalog Folder script, ISPAC files are re-executable. When 
an ISPAC file is re-executed, a new version of the SSIS project is deployed 
to the target SSIS catalog. 


Deploying the Literal Overrides and Catalog 
Environment 


SSIS catalog literal overrides and environment scripts are similar because 
both contain values, either literal override values or values for catalog 
environment variables. Value parameters reside at the top of the Transact- 
SQL script generated and may be modified by the operator prior to 
deployment to the target SSIS catalog, as shown in Figure 7-22. 
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1 © -- envConnectionl ENVIRONMENT VARIABLE VALUES -- 
2 
3 |-- Environment Variable Test\envConnection1\ConnectionString 





ymDemo\Dev; Initial Catalog=TestDB;: 





J" 

Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo_Demo\Test\\4_vmDemo-Demo_SSISDB_ 
1@ | Generated From Catalog Instance: vmDemo\\Demo 

11 Catalog Name: SSISDB 

12 | Folder Name: Test 

13 | Environment Name: envConnection1 

14 Generated By: VMDEMO\A. Ray Leonard 

15 | Generated Date: 8/8/2017 10:05:38 PM 

16 | Generated From: CatalogBase v2.0.2.@ executing on: VMDEMO 

17 | */ 

18 

19 print ‘Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo_Demo\Test\\4_vmDemo-Demo, 
28 | Generated From Catalog Instance: vmDemo\\Demo 

21 | Catalog Name: SSISDB 


Figure 7-22. Editing an SSIS catalog environment script prior to 
execution 


As with the SSIS Catalog Folder script, the Catalog Environment script 
provides rich feedback messages, as shown in Figure 7-23. 
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Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo Demo\ 
Generated From Catalog Instance: vmDemo\Demo 

Catalog Name: SSISDB 

Folder Name: Test 

Environment Name: envConnectionl 

Generated By: VMDEMO\A. Ray Leonard 

Generated Date: 8/8/2017 10:05:38 PM 

Generated From: CatalogBase v2.0.2.0 executing on: VMDEMO 


Deployed to Instance: VMDEMO\DEV 
Deploy Date: 08/09/2017 07:04:43 
Deployed By: VMDEMO\A. Ray Leonard 


Check for folder: Test 
- Test folder exists. 


Environment SSISDB\Test\envConnectionl 


Figure 7-23. Messages from the Catalog Environment Script 
execution 


Please note the Catalog Environment script first checks for the 
existence on the target catalog folder and provides feedback in message on 
its existence. If the target folder does not exist, the Catalog Environment 
script will fail. Catalog environment variables are created in the Catalog 


Environment script. 


Deploying Project and Package References 


Project Reference scripts check for the existence of the catalog folder, 
catalog project, and catalog environment, as shown in Figure 7-24. 
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| print "Check for folder: Test ' 
EJIf Not Exists(Select name 
| From SSISDB. [catalog]. folders 
| Where name = N'Test' ) 
E begin 
set @ErrMsg = ` - Test does not exist. ' 
raisError(@ErrMsg, 16, 1) 
return 
| end 
Else 
E begin 
| print ' - Test folder exists. ' 
| end 
| print '' 
| print "Check for project: DILMSample ' 











-lIf Not Exists(Select name 

From SSISDB.[catalog].projects 

Where name = N‘DILMSample’ ) 

E begin 

set @ErrMsg = ' - DILMSample project does not exist. ' 
raisError(@errMsg, 16, 1) 

return 

| end 

Else 

=) begin 

| print " - DILMSample project exists. ' 

end 

| print 





i 


print "Check for environment: envConnectioni 
FlIf Not Exists(Select name 

| From SSISDB. [catalog]. environments 
Where name = N'envConnection1' ) 














E begin 
set @ErrMsg = ` - envConnectionl environment does not exist. ' 
raisError(@ErrMsg, 16, 1) 
return 

| end 


Figure 7-24. The Project Reference script checks for catalog folder, 
project, and environment 


When executed for the first time, the Project Reference script returns 


messages similar to those shown in Figure 7-25. 
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Be Messages 


Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo Demo\TestG 
Catalog Name: SSISDS 

Folder Name: Test 

Project Name: DILMSample 

Reference Name: Test/envConnectionl 

Environment Name: envConnectionl 

Generated By: VMDEMO\A. Ray Leonard] 

Generated Date: 8/8/2017 10:05:38 PM 

Generated From: CatalogBase v2.0.2.0 executing on: VMDEMO 


Deployed to Instance: VMDEMO\DEV 
Deploy Date: 08/09/2017 19:46:38 
Deployed By: VMDEMO\A. Ray Leonard 


Check for folder: Test 
- Test folder exists. 


Check for project: DILMSample 
- DILMSample project exists. 


Check for environment: envConnectionl 
- envConnectionl environment exists. 


Reference SSISDB\Test\DILMSample\[(.iTest/envConnection1) 
=- Creating Reference SSISDB\Test\DILMSample\[.|iTest/envConnection1) 
- Reference SSISDB\Test\DILMSample\[(.iTest/envConnectionl) created 


Figure 7-25. Project Reference script feedback 


If the Project Reference script is re-executed, the messages reveal that 


the project reference already exists, as shown in Figure 7-26. 
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BE Messages 
Script Name: C:\Users\A. Ray Leonard\Documents\DILM\vmDemo Demo\TestGenerate 
Catalog Name: SSISDB 
Folder Name: Test 
Project Name: DILMSample 
Reference Name: Test/envConnectionl 
Environment Name: envConnectionl 
Generated By: VMDEMO\A. Ray Leonard 
Generated Date: 8/8/2017 10:05:38 PM 
Generated From: CatalogBase v2.0.2.0 executing on: VMDEMO 


Deployed to Instance: VMDEMO\DEV 
Deploy Date: 08/09/2017 19:47:00 
Deployed By: VMDEMO\A. Ray Leonard 


Check for folder: Test 
- Test folder exists. 


Check for project: DILMSamle 
- DILMSample project exists. 


Check for environment: envConnectionl 
=- envConnectionl environment exists. 


Reference SSISDB\Test\DILMSample\(.|Test/envConnection1) 
- Reference SSISDB\Test\DILMSample\[.|iTest/envConnectionl] already exists. 


Figure 7-26. Re-executing the Project Reference script 


The Package Reference script is most complex among the scripts 
generated by SSIS Catalog Compare. As shown in Figure 7-27, the Package 
Reference script includes checks for catalog folder, project, environment, 


and reference. 
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91 & begin 

a2 print " =- Creating Reference 552506 Test \OILMSample\OILMSasple\[ . | Test/envConnectioni ]" 
93 | Declare @reference_id bigint 

o4 = Exec [550508]. [catalog] .[create_environment_reference} 


95 @environment_name = N*envConnection1* 

| » Bréeference_id = Breference_id OUTPUT 

oF | » Beroject_name = N"DILMSenple* 

a8 » Bfolder_neame = N' Test’ 

79 » Benvironment_folder_name = NULL 

160 | » @reference_type = R 

10l | print * - Reference SSISDB\Test\DILMSasple\DILMsample\[.|Test/envConnection1) created" 
Lat end 

103 | else print * - Reference 5S1S06\Test\DILMSample\OILiSanple\(.|Test/envconnectionl) already exists. * 
194 | 

15 


16 | print '' 
lo? == Reference Project Parameter Mapping CM. vebemo\Demo. Test08l .connectionString-->ConnectionString 
186 | print ' - Reference Project Parameter Mapping CM.vebemo)\Demo. TestO8l .cConnectionString-->ConnectionString’ 





169 | print ' - Clear Reference Package or Project Parameter Mapping CH. vebemo\Demo. TestDhl.ConnectionString-->ConnectionString* 
118 Exec [SSIS08)].[catalog].[clear_object_parameter_value] 

177 | Bobject_type = 38 -- package 

112 , Bobject_mame = N"SimplePackage tex" 

113 , Sparameter_name = NOM. vmDemo Demo. TestDB1. ConnectionString" 

114 , folder_name = N'Test' 

115 | , Bpreject_nage = N'DILMSample' 

116 


117 | print " - Reference Project Parameter Mapping CM. veDemo Demo. TestiGl.ConnectionString=->Connectionitring cleared 
115 «| print "" 


119 | print © - Add or Update Reference Package or Project Parameter Mapping CM. vmOemo Demo, TestOGl.Connectionstring-->Connectionstring~ 
120 Exec [S51508). [catalog]. [set_object_parameter_value] 

121 | Bobject_type = 30 -- package 

132 , Bparemeter_name = CM. vebemo Dems. TestObl.connectionstring* 

123 , Bobject_name = N'SimplePackage tex" 

124 , Bfolder_mame = H*Test* 

125 | , Bproject_name = N"DILMSample' 

126 , Bparameter_value = H'ConnectionsString" 

127 | , Bvalue type = 'R' 

12 | 


128 |print ' + Reference Project Parameter Mapping CM. vemo Demo, TestO. Connect ionstring-->ConnectionString added / updated" 
130 | print "' 


Figure 7-27. Existence checks in the Package Reference script 


The Package Reference script always clears the existing value in the 
reference mapping before setting it. 

As one might imagine, there are several messages returned from 
execution of the Package Reference script and they are shown in 
Figure 7-28. 
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Deployed to Instance: VMDEMO\DEV 
Deploy Date: 08/09/2017 19:49:29 
Deployed By: VMDEMO\A. Ray Leonard 


Check for folder: Test 
= Test folder exists. 


Check for project: DILMSamle 
= DILMSarple project exists. 


Check for environment: envConnecticni 
=- envConnectionl environment exists. 


Reference SSISDS\Test\DILMSampie\ DILMSazmle\([.iTest/envConnecticnl) 
=- Reference SSISDS\Test\DILMSarmple\DIiiMSample\[(.iTest/envConnectionl] already exists. 


- Reference Project Parameter Mapping CM.vmDemo\Dem.TestD81 .ConnectionString-->ConnectionString 
- Clear Reference Package or Project Parameter Mapping CéM.vmDemo\Demo.TestD81 .ConnectionString-->ConnectionString 
- Reference Project Parameter Mapping CM. vmDemo\Dem.TestD81.ConnectionString-->ConnectionString cleared 


- Add or Update Reference Package or Project Parameter Mapping CM. vmDermo\ Dero. TestDSi .ConnectionString-->ConnectionString 
- Reference Project Parameter Mapping CM.vmDem\Ders.TestDS1 .ConnectionString-->ConnectionString added / updated 


Figure 7-28. Message returned from Package Reference script 
execution 


Testing with SSIS Catalog Compare 


Return to SSIS Catalog Compare and right-click the Compare button to 
“Refresh Both TreeViews and Compare,’ as shown in Figure 7-29. 


AC $515 Catalog Compare v2.0.2.0 


View Help 


Catalog 1 - 
SQL Server Instance:|vmDemo\Demo | | Refres 


E+ vmDemo Demo 
S-J SSISDB B-)% SSISDE 
E-i Catalog Properties HGP Catalog Properties 
EH Framework fe} Framework 
at Test fH) Test 








Figure 7-29. Refreshing both TreeViews and comparing 
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What’s this? Didn’t I just walk through deploying all these scripted 
artifacts to the target SSIS catalog? Why doesn’t Figure 7-30 show 
everything matching? Please remember you updated the value of the 
ConnectionString catalog environment variable before you executed the 
Catalog Environment script. Figure 7-30 shows the difference that exists 
between the Test catalog folders in the SSIS catalog instances. 





iv: SSIS Catalog Compare v2.0.2.0 - o x 
View Help 
Catalog 1 Catalog 2 
SOL Server instance: vmDemo\Demo Retresh Gear SQL Server instance: \vmDemo\Dev Refresh Gear 
vm Demo Demo To wnDeno \Cev 
‘A ssisoe T Ssisoée 
O Crracg Popetes 
oe | B Tex 
© Precs = Projects 
h > fa 


Figure 7-30. Different data source values 


Now what? These values are, after all, supposed to be different. It’s bad 
if they're the same, in fact. Fear not. Click View > Options, as shown in 
Figure 7-31. 


| AC SSIS Catalog Compare v2.0.2.0 


—=~ 


























Figure 7-31. Opening SSIS Catalog Compare options 


You can ignore catalog environment variable values by checking the 
checkbox shown in Figure 7-32. 
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SSIS Catalog Compare Options E 
L] Ignore Case Sensitivity (Values Only) 
[/] Ignore Catalog Environment Variable Values 
C Ignore Override Values 
L] Ignore Catalog Properties 
[_] Automatically Expand Differences After Compare 


[C] Hide Data Type 

Log Depth:| Light J] 
Log File Retention Days: 90 v 
a on DS Sai 


[_] Dont Show Folder Deploy Confirmation Dialog 

C] Dont Show Project Deploy Confirmation Dialog 

C] Dont Show Project Overrides Deploy Confirmation Dialog 
[C] Dont Show Environment Deploy Confirmation Dialog 

C] Dont Show Reference Deploy Confirmation Dialog 
Scripting Options 

C] Use SSMS Templates 


Figure 7-32. SSIS Catalog Compare options 


Once catalog environment variable values are ignored, a re-compare 
operation shows that the Test catalog folders in your catalogs match. 
Expanding the envConnection1 catalog environment to view the values 
of the ConnectionString catalog environment variable shows the values 
have not changed; they’re still different, and merely ignored, as shown in 
Figure 7-33. 
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Æ ssis Catalog Compare v2.0.2.0 


View Help 
Catakg 1 Catalog 2 
SQL Server instance: vmDemo\Demo | Gear SQL Server instance: | vmDemo\Dev 














|B envConnection | 0 GEE 
te Connection Stang [Stang]: “Data Source =.\Demo: intial Catalog=1 KA Connection String [Stang]: “Data Source «vmDemo\Dev: intial Cz’ 








Figure 7-33. Catalog environment variable values, ignored 


SSIS Catalog Compare may be purchased at dilmsuite.com/ssis- 
catalog-compare. 

Not discussed in this book is CatCompare, the CLI (Command-Line 
Interface) for SSIS Catalog Compare. Learn more at dilmsuite.com/ 
ssis-catalog-compare. 
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SSIS Framework 


Community Edition 


A best practice in SSIS development is to build small, unit-of-work SSIS 


packages. There are several reasons for this: 


e Decoupling: SSIS is software development and a best 
practice with software development is separation of 
concerns. Separation of concerns is primarily achieved 


by decoupling. One way to decouple SSIS is to build 


small, single-function SSIS packages. 


e Testing: If an SSIS package contains seven data flow 
tasks and the design of a source table changes and 
breaks one data flow task, all tasks in the SSIS package 
should be tested. Fewer data flow tasks means less and 


quicker testing. 


e Support: If all SSIS packages contain the minimum 
number of data flow tasks (optimally one) and a 
package execution fails in the middle of the night, 
on-call support has a pretty good idea where to begin 


troubleshooting. 


© Andy Leonard 2018 
A. Leonard, Data Integration Life Cycle Management with SSIS, 
https://doi.org/10.1007/978-1-4842-3276-7_8 
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While these are good and valid reasons to build SSIS solutions with 
several smaller SSIS packages, following this advice causes other issues. 
One issue is you now have a bunch of SSIS packages that require executing 
in some order. What’s a data integration developer to do? 


SSIS Framework Community Edition 


Consider the SSIS Framework Community Edition, a free and open- 
source solution available at dilmsuite.com/ssis-framework-community- 
edition. SSIS Framework Community Edition allows the execution of 
one or more SSIS packages in a specified execution order by executing a 
single stored procedure and passing it a single argument. For example, I 
can execute a test framework application with the following Transact-SQL 


statement: 


exec custom.execute catalog parent package @application name = 


‘Framework Test 


Continuing my theme of “there’s no free lunch,’ SSIS execution 
frameworks greatly simplify execution commands like the one listed above 
but they create another issue: the need to manage a lot of metadata. SSIS 
Framework Community Edition relies on metadata to build a framework 
application, mentioned earlier. A framework application is a collection 
of SSIS packages configured to execute in a specified order. If you build 
idempotent (re-executable) Transact-SQL that includes print statements 
(to inform you of what the T-SQL is doing) and use any kind of formatting, 
you re looking at 30-40 lines of Transact-SQL per SSIS package. That’s a lot 
of T-SQL. 
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Help for SSIS Catalog Projects Already Deployed 


Perhaps you are reading this and thinking, “That’s awesome, but I have a 
bajillion SSIS packages already deployed to my SSIS catalog. What about 
them?” I wrote a blog post called “Adding an SSIS Application to SSIS 
Framework Community Edition” at andyleonard.blog/2017/07/26/ 
adding-an-ssis-application-to-ssis-framework-community-edition. 
I included a script at the end of that post that uses three parameters 
(Framework Application Name, Catalog Folder Name, and Catalog Project 
Name) and from those three pieces of metadata loads the metadata for a 
new framework application into SSIS Framework Community Edition’s 
metadata tables, as shown in Figure 8-1. 


Use SSISD6 
go 


i 
2 
3 
4 Eideclare @frameworkApplicationName varchar(255) = N'Load AdventureWorks2@14 Stage’ 
5 | declare @catalogFolderName nvarchar(128) = N'Stage' 
6 | declare @catalogProjectName nvarchar(128) = N'AdventureWorks2014_Stage_Loader' 
7 | declare @packageName nvarchar( 260) 
8 |declare @sql nvarchar (4000) 
9 | declare @CrLf char(2) = Char(13) + Char(1@) 

1@ | declare @ApplicationIO int 

11 | declare @PackageID int 

12 | declare @ExecutionOrder int = 10 

13 | declare @ExecFlag bit = 1 


15 -ldeclare curPackages Cursor For 
16 iSelect p.[mame] As PackageName 
17 | From [catalog].packages p 


18 | Join [catalog].projects pr 
19 | On pr.project_id = p.project_id 
2@ | Join [catalog].folders f 


21 |On f.folder_id = pr.folder_id 
22 | Where pr.[mame] = @catalogProjectName 
23 | And f.[name] = @catalogFolderName 


Figure 8-1. Building a framework application from an SSIS catalog 
project 


The script reads the SSIS catalog project metadata shown in Figure 8-2 
and loads the framework application metadata into SSIS Framework 


Community Edition metadata tables in a few seconds. 
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Connect ~ Y "f= Oo 





E & Integration Services Catalogs y 
a Ey ss1S0B 
E E Framework 
=) @& Staging 
=} W Projects 
5 og) AdventureWorks2014_Stage Loader 
= © Packages 

Fl 99_Execute_AdventureWorks2014_Stage_Loader_Staging_Packages.disx 
2, AdventureWorks2014_Stage_dbo_AWBuildVersion_TruncateAndLoad.dtsx 
Fl AdventureWorks2014_Stage_dbo_DatabaseLog_TruncateAndLoad.dtsx 
rm AdventureWorks2014_Stage_dbo_ErrorLog_TruncateAndLoad.dtsx 
Fl AdventurelVorks2014_Stage_HumanResources_Department_TruncateAndLoad.dtsx 
Ea AdventureWorks2014_Stage_HumanResources_Employee_TruncateAndLoad.dtsx 
2) AdventureWorks2014_Stage_HumanResources_EmployeeDepartmentHistory_TruncateAndLoad.dtsx 





Figure 8-2. The SSIS catalog project 


The framework application named “Load AdventureWorks2014 Stage” 
that contains 71 SSIS packages can now be executed with the following 
Transact-SQL statement: 


exec custom.execute catalog parent_package @application name = 
‘Load AdventureWorks2014 Stage’ 


Viewing SSIS Catalog Reports 


You can view the executions of these 71 SSIS packages using the Catalog 
Reporting solution built into SSMS. To view all SSIS package executions, 
right-click the SSMS Object Explorer Integration Services Catalogs node’s 
SSISDB node, hover over Reports, hover over Standard Reports, and click 
All Executions, as shown in Figure 8-3. 
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© E Integration Services Catalogs 
ia ABE 
E ge SOL Serve Active Operations 


Create Folder... 











Customized Logging Level 


Start PowerShell 
Database Upgrade 


Execute in Scale Out... 





Manage Scale Out... 







Integration Services Dashboa 





Delete Custom Reports... ; All Executions 
TE 7 i] All Validations 

Refresh All Executions : £ 

Properties f f All Operations 


AN ae ee 


Figure 8-3. Opening the built-in SSIS catalog reports 


The All Executions report displays and surfaces SSIS package 
execution logs, as shown in Figure 8-4. 


All Executions 
on VMDEMO\MBF at 8/9/2017 10:21:26 PM 
This report provides information about the integration Services package executions that have been performed on the connected SOL Server instance. 


T Fier Start time range: 32017- 8/8/2017: Status: All: (4 more) 


Execution Information 


Failed Running Succeeded Others 

ID è +> Status + Report Folder Name = Project Name =: Package Name +; 
125 Succeeded Overview Al Messages Execution Stage Adventure Woks2014_Sa AdventureWorke2014_Sta 
Performance ge_Loader ge_Sales_Store_TruncaleA 

ind Load dso 
124 Succeeded Cheri All Messages Evpcution Stage Adventure Works2014_ 51a AdventureVorke2014_Sta 
Perfomance ge ge_Sales_SpecialOfferProd 
uct_TruncateAnd Load da 
123 Succeeded Chrerview Al Message Etain Stage Adverture Works2014_ Sa Adventure WWorks2014_Sta 
Pedormancce Je gë- specter Tru 
122 Succeeded Overview Al Messages Execution Stage Adventure Works2014_Sia  AdventureWWorks2014_Sta 
Performance ge_Loader ge_Sales_ShoppingCathe 
m_TruncateAnd Lond chee 
121 Succeeded Chernin Al Messages Evpeution Stage Adventure Woks2014_51a Adventure Vorke2014_Sta 
Peformance ge_Loader ge_Sales_Sales TemtoryHis 





Figure 8-4. The All Executions report 


tory_ Truncate AndLoad dtm 


127 


CHAPTER 8 SSIS FRAMEWORK COMMUNITY EDITION 


To summarize, you supplied three pieces of metadata to a Transact- 
SQL script that built a framework application containing 71 SSIS packages, 
and then executed those 71 SSIS packages by starting a single stored 
procedure and passing it one parameter value. 


Viewing SSIS Framework Community Edition 
Metadata 


As mentioned, there’s quite a bit of metadata required for the SSIS 
Framework Community Edition. The script we used earlier is a nice piece 
of automation for entering SSIS Framework Community Edition metadata, 
but what happens when we want to view the framework applications 
already stored? 

Enter Framework Browser, another free utility from DILM Suite that 
you can download at dilmsuite.com/framework-browser, shown in 
Figure 8-5. 





JO Framework Browser v0.4.1.0 (beta) — m x 


View Help 
5515 Catalog 


>m Cat | |e 


5515 Framework 
EHO Load AdventureWorks2014 Stage A 

ii Pi Application Properties 

~ 24] Adventure Works2014_Stage_dbo_A\WBuildVersion_TruncateAndLoad.dtsx 
> Lm Adventure Works2014_Stage_dbo_DatabaseLog_TruncateAndLoad .disx 

ea Adventure Works2014_Stage_dbo_EnorLog_TruncateAndLoad disx 

~ a4] Adventure Works2014_Stage_HumanResources_Department_TruncateAndLoad.dtsx 
é iW Adventure Works2014_Stage_HumanResources_Employee_TruncateAndLoad .disx 
W- i] Adventure Works2014_Stage_HumanResources_Employee DepartmentHistor_TruncateAndLoad.disx 
E- i) AdventureWorks2014_ Stage _HumanResources_EmployeePayHistory_TruncateAndLoad dsx 
EB- tx] Adventure Works2014_Stage_HumanResources_JobCandidate_TruncateAndLoad dtsx 
E- E Adventure Works 2014_Stage_HumanResounces_Shift_TruncateAndLoad.disx 
[+ Ea) Adventure Works2014_Stage_Person_Address_TruncateAndLoad.disx 


Figure 8-5. The framework application's Load AdventureWorks2014 
stage 
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A framework application is a collection of SSIS packages, called 
application packages in the framework, that execute in a specified order. 
Framework Browser lists application packages in the order they execute. 

If you expand the Application Properties virtual folder, you see 
framework application metadata. Expand the Application Package node 
and the Application Package Properties node to surface application 
package metadata, as shown in Figure 8-6. 


- Framework Browser v0.4.1.0 (beta) 





View Help 
SSIS Catalog 


Catalog Instance:| vmDemo\MBF 











S- ta) AdventureWorks2014_Stage_dbo_AWBuildVersion_TruncateAndLoad.dtsx 
gd Application Package Properties 
~~) Catalog Folder: Stage 
cts, Catalog Project: Adventure Works2014_Stage_Loader 
© Application Package ID: 16 
Æ Execution Order: 10 


Figure 8-6. Surfacing application and application package 
framework metadata 


Framework Browser is another free utility from DILM Suite. 
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Catalog Reports 


You ve looked at the SSMS Catalog Reports solution built into SQL Server 
Management Studio (SSMS) a few times in your journey thus far, most 
recently in the previous chapter. As a data integration architect, I find these 


reports awesome, with a couple caveats: 
1. Icannot select text for copy and paste. 


2. In order to view SSMS Catalog Reports, one must 
install SSMS. 


Regarding the second point, there are people in the enterprise who 
have a legitimate need to view the execution of data integration processes 
in the enterprise Production environment, but have no business having 
the remainder of SSMS installed on their machine, much less with access 
to the enterprise Production environment. SSMS is a fantastic utility for 
managing data and administering all aspects of SQL Server. To install 
SSMS for the sole purpose of granting someone access to the SSMS Catalog 
Reports is overkill. 

Please take a look at Catalog Reports, a free and open-source solution 
that’s part of the DILM Suite, at dilmsuite.com/catalog-reports. 

Catalog Reports are designed to look and feel similar to the SSMS 
version, but they reside in SQL Server Reporting Services (SSRS). Look at 
the Executions Report shown in Figure 9-1. 


131 
© Andy Leonard 2018 
A. Leonard, Data Integration Life Cycle Management with SSIS, 
https://doi.org/10.1007/978-1-4842-3276-7_9 


CHAPTER 9 CATALOG REPORTS 


Fa 
| 
1 


fc Executions: All 


12 


Failed Running Succeeded 


ID 
123 
124 
123 
122 
lzi 
120 
119 


1 


| of 2? 


0 


Status 
Succeeded 
Succeeded 
Succeeded 
Succeeded 
Succeeded 
Succeeded 
Succeeded 


Folder 
Stage 
Stage 
Stage 
Stage 
Stage 
Sage 
Stage 


> 


bl 


96 


© | © fom: 


Others 


Project 

AdventureWorks2014_ Stage Loader 
AdventureWorks2014_Stage_Loader 
AdventureWorks2014_ Stage Loader 
AdventureWarks2014_Stage_Loader 
AdventureWorks2014_Stage_Loader 
AdventureWorks2014_Stage_Loader 
AdventureWearks2014_Stage_Loader 


~ ® a) Find | Next 


Package 

AdventurewWorks2014_Stage_Sales_Store_TruncateAndlLoad.dtsx 
AdventurewWorks2014_Stage_Sales_SpecialOfferProduct_TruncateAndload.dtsx 
AdventureWorks2014_Stage_Sales_SpecialOffer_TruncateAndload.disx 
Adventure Works2014_Stage_Sales_ShoppingCartitem_Truncatesnd oad dtex 
Adventureworks2014_Stage_Sales_SalesTennitanHistory_TruncateAndload.dtsx 
AdventureWorks2014_Stage Sales_SalesTerritory_TruncateAndload.disx 
AdventureWorks2014_Stage_Sales_SalesTaxRate_TruncateAndload.dtsx: 


Figure 9-1. DILM Suite Catalog Reports Executions Report 


DILM Suite Catalog Reports surface the same logs and data shown 


in the SSMS Catalog Reports without requiring the installation of 


SSMS. Because the solution is open source, SSRS developers can edit the 


reports to add corporate logos, apply enterprise color schemes, or include 
additional data. 


Report viewers can select and copy text from their browser displaying 


the reports. There are options for exporting the contents of the report, as 


shown in Figure 9-2. 
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TIFF file 

MHTML (web archive) 
CSV (comma delimited) 
XML file with report data 


Data Feed 


Figure 9-2. Report export options 


The right side of the Executions Report contains links to other reports in 
the solution: Overview, Messages, and Performance (shown in Figure 9-3). 


Package Duration Other Reports Start Time 

Adventuraarks2014_Stage_Sales_Store_TruncatetindLoad dts 1.707 Cuenvigur Mertagas Performance S/P 2017 Mc hG:19 PM 
Adventure arks 0i Stage Sales SpecialOferProduct_Truncatedndload dts 1.635 Overview Mesijas Performance BODOG? L197 PM 
Adverturaviorkst0i4_Stage_Sales_ SpecialOffer Truncated aad dts 1.645 Qwerviaws Mexia a5 Performance S2017 119514 PM 
Adverturevorket0i4_Stage_Sales_ShoppingCartem_ThancateAndLoad.ditsx 1.592 Overviews Mesieges Performance B2017 1019:11 PM 
AcvenbureWorks2014_ Stage _ Saes SalesTerritoryHistory_Truntatedndlosd. tee 1.825 Overviewr Metpages Performance B/S 2007 1 19207 Phi 
AdventureWorks2014_ Stage Sales SalesTerritory_TruncateAndioad diese 1.627 Oyerieur Meteages Performance B/S/P007 1c 19204 Phi 
AcventureVarke2014_Stage Sales SalesTaxkate_TruncateandLload.dtsx 1234 Oveniéyyr Meseeqes Performance B/S/2017 10¢19200 PM 
AdwvenbureiVarks2014_ Stage Sales SalesReason_Trancateéndload.ctex 1.2237 Overdéur Mestages Performance B/S/2007 1018-56 PM 
AdventureVorks2014_ Stage Salet_SalesPersonQuotaHistory_TrunctateAndload dtte 1.271 Overievy Mesieges Performance B/S/2007 1h: 18-54 Phi 


Figure 9-3. More fields in the Executions Report 


Click the Overview link for the top-most SSIS package execution to 
open the Overview Report. The Overview Report surfaces a summary of 
SSIS package execution grouped by execution path (individual executables 
in the SSIS package) in the table on the left side of the report. This table 
is useful for determining the longest-running part of your SSIS package. 
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The Execution Information table displays operational log data such as 


SSIS package execution status, duration, start time, and the name of 


the catalog environment referenced for this execution. The Execution 


Parameters table displays SSIS catalog execution parameter settings and 


any parameters overridden or referenced for this execution, as shown in 


Figure 9- 


4. 


Æ Overview 


Themen: Stage deni Werk 4 Stee Lender Aena a Deiee Shore unaoa dies started: B/G? 10679°19 P 


Enia Eedauminss Alaaaaged 


Berorien Line riage 
Result Ceurstion Package Name Task Mame Execetion Path 

Barrene DIN Ainan E E Tan iba a H age 6 enara 4_ a e Tru 
daid dia ak Sere Tiatia aietedlaad 

Sectoid DIA Ainte ge Sales Stor TrescateAs OFT Laid bidii L aga Sale Shere Trume 
daid er ddvnbrweiat JOLA shige 5 JbhirglndpET Lad 

ales Stone deerme ioka bi Se Sale Shore 

Sucoreded OS kirta Se Sales Sooee_Troacaieas Tae Tireni Seaee dukes Snore True 

Aiid dat Adeeb HOLA Shige Ss abd | Toe ae 


aber Sina dara erg 4 Stage sae ieee 
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(STHCHMOUHLLEC L 





Figure 9-4. DILM Suite Catalog Reports Overview Report 


The DILM Suite Overview Report includes navigation links at the top 


of the report to facilitate navigation to the Executions, Performance, and 


Messages reports. Click the Messages link to view the Messages Report, as 


shown in Figure 9-5. 


x Messages 


Messages: Stage\AdventureWorks2014 Stage Loader\AdventureWorks2014 Stage Sales Stor: 


Time 


TruncateAndLoad.dtsx started: 8/9/2017 


Source 


8/9/2017 10:19:21 PM AdventureWorks2014_Stage_Sales_Store_TruncateAndLoad:Finished, 10:19:21 AdventureWorks2014_Stage_Sales_Sto 


PM, Elapsed tme: 00:00:00.375 


re_TruncateAndLoad 


8/9/2017 10:19:21 PM OFT Load AdventureWorks2014_ Stage Sales_Store:Finished, 10:19:21 PM, Elapsed OFT Load 


time: 00:00:00.109. 


AdventureWorks2014_Stage_Sales_Sto 
re 


8/9/2017 10:19:21 PM DFT Load AdventureWorks2014_Stage_Sales_Store:Information: Post Execute DFT Load 


phase is beginning. 


AdventurewWorks2014_Stage_Sales_Sto 
re 


8/9/2017 10:19:21 PM DFT Load AcventureWorks2014_Stage_Sales_Store:Information: "Write DFT Load 


AdventurewWorks2014 Stage Sales Store” wrote 701 rows. 


AdventurewWorks2014 Stage Sales Sto 
re 


Figure 9-5. DILM Suite Catalog Reports Messages Report 
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The Messages report lists every message logged by the execution of the 
SSIS package, when it was logged, the event that raised the message, and 
the name of the executable that raised the event. 


The Performance report displays a graph of execution durations for 
previous successful executions, as shown in Figure 9-6. 


Æ Performance 
Performance: Framework\Framework\Parent.dtsx started: 8/9/2017 10:12:44 PM 


Parent.dtsx Execution Time 


= duraton 


Run Time (seconds) 
mn 
5 





9 22 54 
Execution ID 


Figure 9-6. DILM Suite Catalog Reports Performance Report 


The DILM Suite Catalog Reports solution is free and open-source, and 
addresses a couple issues with SSMS Catalog Reports. 
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BimlExpress Metadata 
Framework 


Business Intelligence Markup Language (Biml) is an XML-based language 
that increases data integration developer productivity and improves SSIS 
code quality. BimlExpress is free and integrates into SQL Server Data Tools 
(SSDT). The BimlExpress Metadata Framework is designed to encourage 
thinking about the possibilities when one combines the power of Biml with 
metadata. 

You can learn more about Biml at varigence.com/biml and at 
bimlscript.com. You can obtain BimlExpress at varigence.com/ 
bimlexpress. You can obtain a copy of the BimlExpress Metadata 
Framework project at dilmsuite.com/biml-express-metadata- 
framework. 

When BimlExpress is installed, a developer can access the BimlExpress 
menu in SSDT. The first document a developer should open is the 
README file, shown in Figure 10-1. 
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Figure 10-1. Viewing the BimlExpress Metadata Framework project 
in SSDT 


The instructions in the README file walk a developer through the 
steps required to use the BimlExpress Metadata Framework. I will follow 
most of those steps in this chapter. You should follow them, too, if you 
desire to use the BimlExpress Metadata Framework. 


Downloading and Installing BimlExpress 


The first step is to download BimlExpress from varigence.com, as shown in 
Figure 10-2. 
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= BimlExpress 


Get more out of Visual Studio with BimlExpress. Fully functional with all versions of SQL Server 


E Free VSIX Download 





Pecrosctt Vue! Nude 
CO98UG TEAM 555 TOOLS TEST è ARXHTECIUR GMLIDPRESS «= ANALYZE «= WINDOW HEP 
-9x 


Sokton Exploeer 


ð o-2 0d _ 


FRE EOT WEW PROACT PWO 






template tier«-"2” 
2- <Bin] xelase"http hemes .varige 
Packages piore 
Package Nawe-" Master tralntMode-"Parallel 5 Scksion Test@enspress (1 project) 
Tasks 
foreach ar table in SootNode. Tables) 
ExecutesQt Namen Create - Connect Lontiane-" Target” 
DirectIin = pectInout E $505 Pochages 
a 4 E Mecelencous 
= Create codPackaget bimi 
t Degicy Target Tables bernd 
* Create TedteMetedete deni 
© Cn a) 


1 Add New Beni Fie 
= Generste 95 Peckages &@ 
Check Bernd For brons 

O Open 


Open With.. 





ope to Ths 
New Schution Explorer View 


Exchuche From Proyect 


x Cut 


Figure 10-2. Downloading BimlExpress 


When the download is complete, execute the .vs1x file to install 


BimlExpress, as shown in Figure 10-3. 


[k] bimlexpress.vsix 


Figure 10-3. Installing BimlExpress 


Accept the License Agreement, as shown in Figure 10-4. 
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[k] VSIX Installer 


BimlExpress 


Select the product(s) you want to install the extension to: 
Microsoft Visual Studio Community 2015 


This extension does not contain a digital signature. 


Varigence, Inc. A 
End User Software License Agreement 

NOTICE: THIS AGREEMENT CONTAINS IMPORTANT TERMS AND 
CONDITIONS THAT REQUIRE YOUR COMPLIANCE. CAREFULLY READ 

THE ENTIRE DOCUMENT. 

This Agreement describes the license terms between Varigence, Inc. 


By clicking “Install”, you agree with the 
above license terms (if any). 














Figure 10-4. Accepting the License Agreement 


Click the Install button to install BimlExpress. 
When the installation completes, the VSIX Installer will display a 
notification similar to that shown in Figure 10-5. 


[E VSIX Installer x 


Installation Complete 


Your extension has been successfully installed. Please close and 
restart all target application instances for changes to take effect. 


© Microsoft Visual Studio Community 2015 





View Install Log 


Figure 10-5. BimlExpress installation is complete 
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Click the Close button and open SQL Server Data Tools (SSDT). 


Click Tools > Extensions and Updates, as shown in Figure 10-6. 


J Start Page - Microsoft Visual Studio 
File Edit View Debug Team Tools | Test  BimlExpress Analyze Wii 


0-0 | D- a h aP | D -| "@ Connect to Database... 
oes "= Connect to Server... 
SOL Server 





[] Code Snippets Manager... 


aloja Jaag 


Visual Studio 


Choose Toolbox Items... 






3 NuGet Package Manager 

S [Si Extensions and Updates... 

tA start Create GUID 

S New Project... Error Lookup 

= Open Project... Preemptive Protection - Dotfuscator 





Figure 10-6. Opening Extensions and Updates 


The Extensions and Updates dialog is where you manage BimlExpress 
and other SSDT plugins, as shown in Figure 10-7. 


| Extensions and Updates T x | 
4 Installed Sort by: Name: Ascending = Search Installed (Ctrl=E) P- 
Ali &> Azure AD Authentication Connected Service = Created by: Varigence 
Controls Provides the wizard to configure Azure AD Authentication Single Sign-On 


Date Installed: 2/10/2017 


Samples in web projects. Version: §.0.61915.0 
Templates BimlExpress |] Automatically update this extension 
SDKs =. 





Bm ESpress a Yeun sudo edhension TE 


Tots building Birni files. 
t Online aa ei 
-H Developer Analytics Tools 
K Epara WE Telemetry search, CodeLens integration, and other tools for Application 


Insights. App deployment tools for HockeyApp. 


Figure 10-7. The Extensions and Updates dialog 


If you see the BimlExpress menu item shown in Figure 10-8, 
BimlExpress is installed and enabled. 
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( BimlExpress)) l 


Figure 10-8. BimlExpress installed and enabled 


Downloading BimlExpress Metadata Framework 


Let’s next download the BimlExpress Metadata Framework code from 
dilmsuite.com/biml-express-metadata-framework, as shown in 
Figure 10-9. 


Bin! Expreva brudeta F E 


& C7 | @ Secure | https dilmaatecomdownloads/bml-eopress-metadate- framework; 


A Welcome Contact Us MailingList ë Q 


Biml Express Metadata Framework 
D Pitna 1. 21 B ancy 
The Bimi Express Metadata Framework ues Business mlelizenmce Markun Language (Bini) moc 


= Reed metadate stored in a SQL Server database. 
* Build So piget that cree he anget database, Schemas, ad Wie. 


+ Build an $505 progect that contains one $505 package per table, pius à Controller package that executes each table-package 


You can beam mare by reading Ben! Express Metadata Framework aft SOLBiog.com or by viening the recording of the webinar Save Time aid improve S515 Quality with Bim. 





Figure 10-9. Downloading BimlExpress Metadata Framework 


The BimlExpress Metadata Framework file is a text file named 
BimlExpressMetadataFramewor.renametozip.txt, shown in Figure 10-10. 


Name 
Biml Express Metadata Framework 
BimlExpressMetadataFramework.renametozip 


Figure 10-10. The BimlExpress Metadata Framework file 
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One the file is downloaded, rename it as shown in Figure 10-11. 


Name 


|_| BimlExpressMetadataFramework.renametozip.tt 


ae 


Date modified Type 


8/10/2017 5:15PM __Text Document 
Open 

Print 

Edit 

7-Zip > 
CRC SHA > 
Edit with Notepad++ 

Scan with Windows Defender... 

Open with 


Share with 


Restore previous versions 





Send to > 
Cut 
Copy 


Create shortcut 
Delete 


Rename 


Properties 


Figure 10-11. Preparing to rename the Biml Express Metadata 


Framework Download file 


Changing the file extension to “zip” will trigger the warning shown in 


Figure 10-12. 
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Rename 


A If you change a file name extension, the file might become unusable. 


Are you sure you want to change it? 


No 


Figure 10-12. Warning about renaming a file and changing the 
extension 


After renaming, the Biml Express Metadata Framework Download file 
appears as shown in Figure 10-13. 


$| [A |) = | Downloads 


Home Share View 


db 4 « vmDemo_C (C) > Users > A. Ray Leonard > Downloads > 


h 


Name 
wr Quick access : 
E BimlExpressMetadataFramework.zip 


E This PC 


Figure 10-13. The renamed Biml Express Metadata Framework 
Download file 


Right-click the Biml Express Metadata Framework file and click Extract 
All, as shown in Figure 10-14. 


p! BimlExpressMetadataFramework.zip 


Open 
Open in new window 


Extract All... 


Figure 10-14. Preparing to extract the Biml Express Metadata 
Framework file 
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The Extract Compressed (Zipped) Folders dialog displays as shown in 
Figure 10-15. 


E Extract Compressed (Zipped) Folders 


Select a Destination and Extract Files 


Files will be extracted to this folder: 


| E:\Projects\BimlExpressMetadata Frameworky1 


[4] Show extracted files when complete 


Cancel 


Figure 10-15. Choosing a folder for the extracted files 


Click the Extract button to extract the compressed files. 

After the contents of the zip file have been extracted, browse to the 
folder and open the BimlExpressMetadataFramework.sln file, as shown in 
Figure 10-16. 
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W.. > BimlExpressMetadataFramework > v & Search BimlExpressMetadataF... J 
A 
Name Date modified Type 
iws 1/31/2017 8:39AM File folder 
E BimlExpressMetadataFramework 1/31/2017 11:15AM _ File folder 
fa BimlExpressMetadataFramework.sin 8/10/2017 5:22 PM Microsoft Visual S... 


5] BimlExpressMetadataFramework.v12.suo 8/10/2017 5:22 PM Visual Studio Solu... 


| debug.log 8/10/2017 5:22 PM Text Document 


Figure 10-16. Opening the BimlExpressMetadataFramework.sln file 


Following the README File Instructions 


At the time of this writing, the README file included with the 
BimlExpressMetadataFramework project reads as follows: 


BimlExpressMetadataFramework Notes 


0. Download and install BimlExpress (https://varigence.com/ 
BimlExpress) 
1. Restore the BimlMetadata database. 
2. Update the connection string in params.biml to aim it at 
your BimlMetadata database. 
3. Update connection strings in the di.Connections table aim 
them at your AdventureWorks2014 database. 
4. In Solution Explorer, multi-select the following Biml files 
in the Miscellaneous virtual folder to create the destination 
Database: 

- 0 00 BuildConnections.biml 

- 0 10 BuildDestinationDatabaseMain.biml 
4a. Right-click in Solution Explorer (or dropdown the 
BimlExpress menu) and click Generate SSIS Packages. 
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5. In Solution Explorer, multi-select the following Biml files 
in the Miscellaneous virtual folder to create the destination 
Schemas: 
- 0 00 BuildConnections.biml 
- 0 20 BuildDestinationSchemasMain.biml 
5a. Right-click in Solution Explorer (or dropdown the 
BimlExpress menu) and click Generate SSIS Packages. 
6. In Solution Explorer, multi-select the following Biml files 
in the Miscellaneous virtual folder to create the destination 
Tables: 
- 0 00 BuildConnections.biml 
- 0 30 BuildDestinationTablesMain.biml 
6a. Right-click in Solution Explorer (or dropdown the 
BimlExpress menu) and click Generate SSIS Packages. 
7. Execute the packages: 
- 10 Build Destination Databases.dtsx 
- 20 Build Destination Schemas.dtsx 
- 30 Build Destination Tables.dtsx 
8. Delete all Project Connection Managers and Packages. 
9. In Solution Explorer, multi-select the following Biml 
files in the Miscellaneous virtual folder to create the SSIS 
Packages: 
- 0 00 BuildConnections.biml 
- 0 10 BuildDestinationDatabaseMain.biml 
- 0 20 BuildDestinationSchemasMain.biml 
- 0 30 BuildDestinationTablesMain.biml 
- 1 99 Main.biml 
9a. Right-click in Solution Explorer (or dropdown the 
BimlExpress menu) and click Generate SSIS Packages. 


We have completed Step 0 already. 
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Restoring the BimlMetadata Database 


In SSMS Object Explorer, right-click the Databases node and click Restore 
Database, as shown in Figure 10-17. 





| E vmDemo\Demo (SQL Server 13.0.4206.0 - VMDEMO\ 
=) e ieee 











a É Syster New Database... 
a E Datab Attach... 
o W Otot — 


o” real _ Restore Database... 
Figure 10-17. Preparing to restore the BimlMetadata database 


Select the Device option for Source and click the ellipsis to select 
the backup file. Navigate to the file system folder you extracted 
the BimlExpressMetadataFramework. zip file from and select the 
Bim|Metadata database backup file, as shown in Figure 10-18. 
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W Locate Backup File - VMDEMO\DEMO oO x 

Backup Fie location: E\Projects\BimlEpressMetadataFrameworo] | @ | | Mi: 
B-E OToBiml_London_Jun2017 ~ || lll BimlExpressMetadataFramework | 
a- OTOSSIS ite] Eimi Metadata 20170131 bak 





B AndysDumb Script Task Test 
| E-H ATLPreconJan2016 
| fl Ban 
: G-A Bl Batch 
H-E Biml_Test 
: W- BeniBook 
| E- BimlBookChapter12 
H-E BenlBookChapter7 
: (00) BimlBookChapter7_2 
0 BimiDemos 
| (0) BimlExpressMetadataFramewor 
| 0) BimlExpressMetadataFramewor 
=| BimlExpress MetadataFramewor 
G-E BimlEnressMetadata Frame 
| H- BimlEpressMetadataFi ., 
€ > 


File name: (BimiMetadata_20170131 bak | Backup Files("bak:"tm:"log) vi 


OK Cancel 





Figure 10-18. Selecting the BimlMetadata database backup file 


Click the OK button to select the BimlMetadata database backup file 
and return to the Restore Database dialog, as shown in Figure 10-19. 
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Sz Restore Database - BimiMetadata 











= o x 
ü Ready 
Select a page C Script ~ | @ Help 
# (General 
E ies 
E Odim Sourte 
©) Database: 
© Device: [E Projects) Bimi Exoress MetadataFramework 1\BrriExpressMetadataFramework Berl] l 
Database: ‘BemiMetadata w] 
Database: Beiter 
Restore to: ‘The last backup taken (Tuesday. January 31, 2017 12:17:45 PM) | | Timeline... | 
Restore plan 
Backup sets to restore: 
Resore Mame Component Type Sewer Database Postion First | 
| BinlMetedata-Ful Database Backup Database Full VMSGLIADEMO BiriMeadata 1 4600 
Connection 
yF vm Demo Demo [VMDEMOrA. 
Ray Leonard] 
Vi 
Progress 
£ > 
o 
Verty Backup Meda 


| Cancel || Help 
Figure 10-19. BimlMetadata backup file selected 


Click the Files page and check the “Relocate all files to folder” 
checkbox. Navigate to the Data and Log file folders, and double-check that 


the “Restore As” paths match your selections for Data and Log file folders, 
as shown in Figure 10-20. 
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Sis Restore Database - BimiMetadata 


Select a page | {J Script ~ | @) Help 


# Fics Restore database files as 








(Connection 
vm Demo Demo [VMDEMO>A 
w Ray Leonard] 





Figure 10-20. Configuring restore file locations 


On the Options page, check the “Overwrite the existing database 
(WITH REPLACE)” checkbox, as shown in Figure 10-21. 
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ije Restore Database - BimiMetadata 





= B 
ib Ready 
Select a page |E] Script ~ | @ Help 
# General 
E Files 
F Oot Restore options 
E] Overte the existing database (WITH REPLACE) 
[C] Preserve the replication settings (WITH KEEP_REPLICATION) 
[C Restrict access to the restored database (WITH RESTRICTED_USER) 
Recovery slate: RESTORE WITH RECOVERY 
Standby file: E‘\Frogram Files‘ Mroeoft SOL Servern MSSOLTI DEMONMS SOL Backup, 
Leave the database ready to use by oing bock uncommitted tansactions. Additional trancechon logs cannot be restored. 
Tad-Log backup 
Backup file: E) Program Files\Microsoft SOL SenrerWMS 501 13.0 EMO *\wMS5OL\ Backup’, 
Server conmections 
Prompt 
CI Prompt before restoring each backup 
[Gonnection ean, The Full-Text Upgrade saver progerty controis whether fullest indexes are imported, buk, or reset for the restored 
qi vmnDemo Demo [VMDEMO‘A. D databace. 
Hay Leonard] 
P 


Figure 10-21. Configuring the restore to overwrite the existing 
database 


Click the OK button to begin the Database Restore operation. 


If all goes as planned, you should see the message box shown in 
Figure 10-22. 
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Microsoft SQL Server Management Studio xX 


D Database 'BimIMetadata' restored successfully. 


Figure 10-22. Successful restore! 


Refresh Object Explorer. You should now see the Bim|Metadata 
database listed, as shown in Figure 10-23. 


Obj ect Exp orer 





Connecty ¥ *# © * 
= k vmDemo\Demo (SOL Server 13.0.4206.0 
= © Databases 
& © System Databases 
Œa © Database Snapshots 
= fg 0.to_Biml_London 
& Æ AdventureWorks2014 
© Æ BimIBook 






Figure 10-23. The BimlMetadata database 


If you do not have a copy of the AdventureWorks2014 sample database, 
search for a download of the Microsoft sample database, download it, and 
restore or otherwise build it. At the time of this writing, you may obtain 
a database back file of AdventureWorks2014 at msftdbprodsamples. 
codeplex. com/downloads/get/880661, but Microsoft plans to shut down 
CodePlex and migrate the code to GitHub. 
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Updating the Connection String Variable Value 
in the Params.biml File 


The next step in the README file is to update the connection string in the 
params .biml file to aim it at your BimlMetadata database, as shown in 
Figure 10-24. 





ç ç y$ [ 
sport namespaces" Systes.Data.SalClient H 


nnectionsAdoMetadataConnectionString = "Data Source-vaDesco\Deso; Initial Catalog-BimlMetadata;Integrated Security=True;”; 


Figure 10-24. Updating the BimlMetadata connection string in 
Params.biml 


Updating the di.Connections Table 


The next step in the README file is update connection strings in the di. 
Connections table to aim them at your AdventureWorks2014 database, as 
shown in Figure 10-25. 


154 


CHAPTER 10 BIMLEXPRESS METADATA FRAMEWORK 


=Select * From di.Connections 


1 
2 
3 E\Update di.Connections 

4 |Set ConnectionString = Replace(ConnectionString, 'vmSql14\', 'vmDemo\') 
5 

6 

7 


| Select * From di.Connections 


100% ~ 

333 Results B Messages 

ConnectionlD ConnectionName Connection String 
1 [1 | WideWoddimportters_Source data source=vmSql14\Demojnitial catalog=WideWor... 
2 2 — WideWorldimporters_Stage data source =vmSqi14\Demo: intial Catalog=WideWor... 
3 3 __master__WideWordimporters_Stage data source=vmSqi14\Demo:initial Catalog=master:Pr... 
4 4 Adventure Works2014_Source data source =vmSqi14\Demo initial catalog=Adventur... 
5 5 Adventure Works2014_Stage data source =vm Sql 14\Demo: Initial Catalog=Adventur... 
6 6 __master_ AdventureWorks2014_Stage data source=vmSqi14\Demo: initial Catalog=master:Pr... 
4 

ConnectioniID ConnectionName Connection String 


ree re rey 


1 | WideWorldimporters_Source data source=vmDemo\Demo jinttial catalog=Wide Worldimporters pre 
2 2 — WideWorldimporters_Stage data source=vmDemo\Demo:|ntial Catalog=WideWordimporters_St 
3 3 __master__WideWorldimp... data source=vmDemo\Demo: Initial Catalog =master: Provider=SQLNC 
4 4 Adventure Works2014_So... data source=vmDemo\Demo initial catalog=Adventure Works2014p 
5 5 Adventure Works2014_Sta... data source=vmDemo\Demo: Initial Catalog=Adventure Works2014_ 
6 6 __master_AdventureWor... data source=vmDemo\Demo: initial Catalog=master:Provider=SQLNC 


Figure 10-25. Updating the Adventworks2014 connection strings in 
the di.Connections table 


Generating the Destination Database 


In the BimlExpressMetadataFramework project, multi-select the 0 00_ 
BuildConnections.biml and0 10 BuildDestinationDatabaseMain.biml 
files. Right-click and click Generate SSIS Packages, as shown in Figure 10-26. 
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À~ 


~ t > 
a F i wv =z 
SOIMUTION EXDi 


Ceoa|\e-sablsr—= 
Search Solution Explorer (Ctri+;) 

4& BimlExpressMetadataFramework (SQL Server 2014) 
@ Project.params 

fa) Connection Managers 

fa) SSIS Packages 
fa) Miscellaneous 









EN ARK P 
O J 00 BuildConnections.bimi 


Generate SSIS Packages 


Check Biml For Errors 


AU 


Open 


Figure 10-26. Building the destination database SSIS package 


When the 0 00 BuildConnections.biml and0 10_ 
BuildDestinationDatabaseMain.biml files execute, they create the 
project connection manager named _master__AdventureWorks2014_ 
Stage and the SSIS package named 10 Build Destination_Databases. 
dtsx, as shown in Figure 10-27. 


La 


SOlUuti p) a i 


cog|o-sé@als— 
Search Solution Explorer (Ctri+;) 


{Æ BimlExpressMetadataFramework (SQL Server 2014) 
& Project.params 
a | Connection Managers 
¥ _master_AdventureWorks2014 Stage.conmgr 
a {al SSIS Packages 
EX 10_Build_Destination_Databases.dtsx 





Figure 10-27. Creating an SSIS package that builds the target 
database 
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Generating the Destination Schemas 


In the BimlExpressMetadataFramework project, multi-select the 0 _00_ 
BuildConnections.biml and0 20 BuildDestinationSchemasMain.biml 
files. Right-click and click Generate SSIS Packages, as shown in Figure 10-28. 





(Com|o-Saa|s— 


| Search Solution Explorer (Ctri+;) 


| {È BimlExpressMetadataFramework (SQL Server 2014) 
&@ Project.params 
4 ‘) Connection Managers 
 _ master_AdventureWorks2014 Stage.conmgr 
4 tm) SSIS Packages 
A 10 Build_Destination_Databases.dtsx 
| 4 tal Miscellaneous 


0 00 BuildConnections.biml 












= Generate SSIS Packages 
CY Check Bim For Errors 
© Open 


Figure 10-28. Building the destination schemas SSIS package 


When the 0 00 BuildConnections.biml andO 20_ 
BuildDestinationSchemasMain.biml files execute, they create the project 
connection manager named AdventureWorks2014 Stage and an SSIS 
package named 20 Build Destination Schemas.dtsx, as shown in 
Figure 10-29. 
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a 





50 ‘olutio nE = Exp 


ea sap] = 


| Search Solution Explorer (Ctri+;) 








| eee BimlExpressMetadataFramework (SQL Server 2014) 
&@ Project.params 

a l Connection Managers 
y _ master_AdventureWorks2014_Stage.conmgr 
Y AdventureWorks2014_Stage.conmgr 

a tm) SSIS Packages 
E) 10_Build_Destination_Databases.dtsx 
EX 20 Build Destination Schemas.dtsx 


Figure 10-29. Creating an SSIS package that builds the destination 
schemas 


Generating the Destination Tables 


In the BimlExpressMetadataFramework project, multi-select the 0 _00_ 
BuildConnections.biml and 0 30 BuildDestinationTablesMain.biml 
files. Right-click and click Generate SSIS Packages, as shown in Figure 10-30. 
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Se eee ee, E 
SOlUtION Explorer 


| @ ©2@/|o-s a a|s = 
Search Solution Explorer (Ctri+;) 


& BimlExpressMetadataFramework (SQL Server 2014) 
& Project.params 
4 a) Connection Managers 
¥ _master_AdventureWorks2014_Stage.conmgr 
@ AdventureWorks2014_Stage.conmgr 
4 SSIS Packages 
E 10 Build_Destination_Databases.dtsx 
E 20 Build_Destination_Schemas.dtsx 
4 ġa Miscellaneous 
DO I 00 Build¢ onnections.biml 
O 0.10 _BuildDestinationDatabaseMain.biml 
O 0.20 BuildDestinationSchemasMain.bim! 
i) 0 30 BuildDest nat on [ablesMain.bim! 
a 1_99_Main.biml = Generate SSIS Packages 
1_BuildConnections.biml è | 
0 2_BuildDatabases.bim! or Ree Pimi Eor Eror 
M 2 RiildSehemac himl C Open 











Figure 10-30. Building the destination tables SSIS package 


The AdventureWorks2014 Stage project connection manager is 
regenerated by the 0 30 BuildDestinationTablesMain.biml file. The 
Confirm Overwritten Items dialog displays as shown in Figure 10-31. 


S Confirm Overwritten items — o x 


in addition to 1 new tems, the template generated the following tems that conflict with exdsting tems. Which of these tems would 
you like to overwrite? 


C) Select all 


E E \Proyects \Bini Express Metadata Framework 1\Bim Express Metadata Framework \Bimi Express Metadata Framework Bim Egr 


[Comme] Cancel 


Figure 10-31. Confirm Overwritten Items dialog 
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Click the Commit button to proceed. 

When the 0_30 BuildDestinationTablesMain.biml file executes 
and overwrites the project connection manager, it also overwrites the file 
named AdventureWorks2014 Stage.conmegr, causing a reload operation, 
as shown in Figure 10-32. 


Microsoft Visual Studio ? x 


E:\Projects\BimlExpressMetadataFrameworkv1\BimlExpressMetadataFramework\BimlExpressMetadataFramewor 
k\BimlExpressMetadataFramework\AdventureWorks2014_Stage.conmgr 


The file has unsaved changes inside this editor and has been changed externally. 
Do you want to reload it? 


[_] Reload modified files unless there are unsaved changes 


(You can change this setting in Tools | Options | Environment | Documents) 


Yes Yes to All No No to All 


Figure 10-32. Reloading the overwritten AdventureWorks2014_Stage 
connection manager file 


Click the Yes button to proceed. 

When the 0 00 BuildConnections.biml and0 30_ 
BuildDestinationTablesMain.biml files execute, they create the project 
connection manager named AdventureWorks2014_ Stage and an SSIS 
package named 30 Build Destination Tables.dtsx. 


Executing the Create-Destination-Artifacts SSIS 
Packages 


Right-click the 10 Build Destination Databases.dtsx SSIS package to 
create the target database, as shown in Figure 10-33. 
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Search Solution Explorer (Ctri+;) P~ | 


4&2 BimlExpressMetadataFramework (SQL Server 2014) 
gë Project.params 
a l| Connection Managers 
@ _master_AdventureWorks2014_Stage.conmgr 
@ AdventureWorks2014_Stage.conmagr 
4 tl SSIS Packages 
E 10_Build_Destination_Databases.dtss 











E, 20_Build_Destination Scher (2) Add New Biml File 
A 30_Build_Destination_Table Ci 
4 tl Miscellaneous i 
0 0_00.BuildConnections.bin “8 Add New VB File 
0 0_10_BuildDestinationDatat Set as StartUp Object 
0) 0_20_BuildDestinationScher Execute Package 


0 0.30BuildDestinationTable EI 
M 1 00 Msin himal | Deploy Package 


Add New C# File 


Figure 10-33. Executing the 10_Build_Destination_Databases.dtsx 
SSIS package 


If all goes as planned, the 10 Build Destination Databases.dtsx 
SSIS package executes successfully, as shown in Figure 10-34. 





1U_Builc tinatio...bases.dtsx [Design] | README _BimlExpres.. 
fq Control Flow v Data Flow @@ Parameters [] Event Handlers 


cp SQL Create AdventureWorks2014 Staging 


Figure 10-34. Successful execution of the 10_Build_Destination_ 
Databases.dtsx SSIS package 


Return to SSMS and refresh the Databases node in Object Explorer, as 
shown in Figure 10-35. 
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Connectty ¥ *¥ © YO- 
© E vmDemo\Demo (SQL Server 13.0.4206.0 - VMDEMO 





New Database... 


Attach... 
Restore Database... 
Restore Files and Filegroups... 





Filter 
= je BMet —— 
w Destin Deploy Data-tier Application... 
w Offic Import Data-tier Application... 
m E Repo Start PowerShell 
W Repo 


© Æ Sourc Reports 








W SSiSincrementalLoad_Source 
H Æ SSiSincrementalLoad_Stage 
W TestDB 

i WeatherData 


Figure 10-35. Refreshing the Databases node in Object Explorer 


The AdventureWorks2014_Staging database now appears in the target 
instance, but no schemas or tables have yet been created, as shown in 
Figure 10-36. 
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= E vmDemo\Demo (SQL Server 13.0.4206.0 
=) F Databases 
E System Databases 
©) Database Snapshots 
g 0_to_Biml_London 
fg AdventureWorks2014 





©) Database Diagrams 
© É Tables 

E System Tables 

© FileTables 

© External Tables 
@ Views 


Figure 10-36. AdventureWorks2014_Staging with no schemas or 
tables 


Right-click the 20 Build Destination Schemas.dtsx SSIS package to 
create the schemas in the target database, as shown in Figure 10-37. 
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cof|o-seapls— 
Search Solution Explorer (Ctri+;) 


4G? BimlExpressMetadataFramework (SQL Server 2014) 
@@ Project.params 
4 a| Connection Managers 
@ _master_AdventureWorks2014_Stage.conmgr 
8 AdventureWorks2014_Stage.conmgr 
4 tw) SSIS Packages 
E 10 Build_Destination_Databases.dtsx 
D 2) Build Destination Schemas.dtsx 
E 30_Build_De |=) Add New Biml File 


4 l Miscellaneous -C# Add New C#File 
















O 0.00_BuildC! | 
D 0_10_BuildD VB Add New VB File 
O 0_20_BuildD) Set as StartUp Object 
O 0_30_BuildD| pee PE 
O enn Deploy Package 


Figure 10-37. Executing the 20_Build_Destination_Schemas.dtsx 
SSIS package 


If all goes as planned, the 20 Build Destination Schemas.dtsx SSIS 


package executes successfully, as shown in Figure 10-38. 
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og BimlExpressMetadataFramework (Running) - Microsoft Visual Studio 

File Edit View Project Build Debug Team Format S515 Tools Test BimlExpress Analyze Window 
f@-o0|8-gewp|9-¢- Continue -| M jn mw O| %| - 

= Process: [14688] DtsDebugHost.exe - M Lifecycle Events ~ Thread: A eA Sta 
stinati...hemas.dtsx [Design] © x MA Semele ELA E 

T cla Date Flow g Parameters [E] EventHanders "= Package Explorer (} Progress 


W 











op SQL Create Person ¥ fac SQL Create Production 


cp SOL Create HumanResources 


Connection Manages 


 (project)__master__AdventureWorks2014_Stage  @ (project) AdventureWorks2014_Stage 


Call Stack Breakpoints Exception Settings Command Window Immediate Window Output ErrorLlist Autos Locals Watch! 





Figure 10-38. Successful execution of the 20_Build_Destination_ 
Schemas. dtsx SSIS package 


Right-click the 30 Build DestinationTables.dtsx SSIS package to 
create the tables in the target database, as shown in Figure 10-39. 


165 


CHAPTER 10 BIMLEXPRESS METADATA FRAMEWORK 


Figure 10-39. Executing the 30_Build_Destination_Tables.dtsx SSIS 
package 





colo- sapla 
| Search Solution Explorer (Ctrl+;) 


¿Æ BimlExpressMetadataFramework (SQL Server 2014) 
&@ Project.params 

4 ‘ml Connection Managers 
@ _master_AdventureWorks2014_Stage.conmgr 
8 AdventureWorks2014_Stage.conmgr 

4 ‘ml SSIS Packages 

E) 10_Build_Destination_Databases.dtsx 

EL 20_ Build Destination- Schemas. dtsx 














Add New Bimi File 


kre 


| Add New C# File 
| VB Add New VB File 
30_ : Set as StartUp Object 
991 | Execute Package 
D 1 Bus Deploy Package 
O 2_Builc 


D 3 Bui © Open 


If all goes as planned, the 30 Build Destination Tables .dtsx SSIS 


package executes successfully, as shown in Figure 10-40. 
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ne | BirnlExpressiMetadataFramework (Running) - Microsoft Visual Studio 

File Edit View Project Build Debug Team Format S515 Tools Test BimlEmpress Analyze Window Help 

/@-9o| -2HP 9 - - | eveloy Continue -| H. $N E 5 EA 1% 

| Process: [15040] DtsDebugHost.exe = Lifecycle Events ~ Thread: - Y e Xt Stack Frame 
inatio,..Tables.dtsx [Design] = = R SA Wish. aa 

wu OstaFlow g Parameters [F] EventHandiers "= Package Explorer È Progress 









w 


fap SQL Create dbo_AWBuildVersion cp SQL Create dbo_DatabaseLog p SQL Create dbo_ErrerLog 


Connection Manages 
@ (project) __master_AdventureWorks2014_Stage (project) AdventureWorks2014_Stage 


Call Stack Breakpoints Exception Settings Command Window Immediate Window Output ErrorList Autos 





Figure 10-40. Successful execution of the 30_Build_Destination_ 
Tables.dtsx SSIS package 


Return to SSMS and refresh the AdventureWorks2014_Staging Tables 
node in Object Explorer, as shown in Figure 10-41. 
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Connecty ¥ *¥ = OY G -= 
5 k vmDemo\Demo (SOL Server 13.0.4206.0 - VMDEMO\A. Ray Leonard) 
= © Databases 

E System Databases 

©) Database Snapshots 

i 0_to_Biml_London 

AdventureWorks2014 

=  AdventureWorks2014_ Staging 

© Database Diagrams 






















p mS New > 
2 = Fil Filter b 
me Ext Sa EE 
H Views Start PowerShell 
É Extern Reports b 
E Synon o | 







É Progre Refresh 


Figure 10-41. Refreshing the AdventureWorks2014_Staging Tables 
node 


The AdventureWorks2014_Staging database now contains schemas 
and tables, as you can see in Figure 10-42. 
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=) $Æ AdventureWorks2014_Staging 
rs = Database Diagrams 


m 
ET 


+ 


ha! 





= pull Tables 

© FileTables 

©) External Tables 

FE dbo.AWBuildVersion 

EE dbo.DatabaseLog 

FB dbo.ErrorLog 

FE HumanResources.Department 

FE HumanResources.Employee 

) FB HumanResources.EmployeeDepartmentHistory 
FE HumanResources.EmployeePayHistory 
EE HumanResourcesJobCandidate 





+) & £ 


+) 


H & & & & & 


(+) 


B9 HumanResources.Shift 

FA Person.Address 

EB Person.AddressType 

+) EB Person.BusinessEntity 

) FB Person.BusinessEntityAddress 
| FB Person.BusinessEntityContact 
| E 
EE 


+ 


r 
L 


+) 


ad +) 


Person.ContactType 
Person.CountryRegion 
© FR Person.EmailAddress 
© FR Person.Password 

&) FR Person.Person 


Figure 10-42. The AdventureWorks2014_Staging database with 


schemas and tables 


Deleting Existing Artifacts 


Delete the existing SSIS packages and project connection managers from 


the SSIS project, as shown in Figure 10-43. 
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ool o5 am| 
| search Solution Explorer (Ctri+;) 
E BimlExpressMetadataFramework (SQL Server 2014) 


&@ Project.params 
a ġa] Connection Managers 







() _master_AdventureWorks2014_Stage.conmgr 
AdventureWorks2014_5tage.conmgr 


fal SSIS Packages 
E 10 Build Destination _Databases.dtsx 
EY 20 Build Destination Schemas.dtsx 
(30. Build Destiy TS 

4 tf Miscellaneous | C Open 
D 0.00_BuildCon <> View Code 
D 0.20 BuildDes Cè View Designer 
C 0.20 BuildD 
DO 0_30_BuildD 
O 1_99_Main.bi Exclude From Project 
O 1_BuildConnec 
D 2_BuildDataba: 
O 3.BuildScheme O° Copy 
D 4 .BuildTablesA X Delete 
D 5_BuildPackag Rename 
C) 6 BuildStaging 
M 7 BuildProiec 








New Solution Explorer View 


Cut 


P Properties 
Figure 10-43. Deleting the existing artifacts 


You do not have to delete the existing artifacts. If you do not delete the 
existing artifacts, BimlExpress will prompt you to confirm overwrites of 
existing artifacts. 


Generating SSIS Packages 


Multi-select 0 00 BuildConnections.biml, 
O 10 BuildDestinationDatabaseMain.biml, 
O 20 BuildDestinationSchemasMain.biml, 0 30_ 
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BuildDestinationTablesMain.biml, and1 99 Main.biml. Right-click 


and select Generate SSIS Packages, as shown in Figure 10-44. 


Solution Explorer 
CO f|o-s ae r= 


| Search Solution Explorer (Ctrl+;) 





4g? BimlExpressMetadataFramework (SQL Server 2014) 
&@ Project.params 
fa) Connection Managers 
fa) SSIS Packages 
a t Miscellaneous 
D 0 00 BuildConnections.biml 
(E) 0 10 BuildDestinationDatabaseMain.bim! 


D 0 20 BuildDestinationSchemasMain.biml 

D 0_30_BuildDestination TablesMain.biml 

D 1_99_Main.biml 

O 1_BuildConnectio = Generate SSIS Packages 
[À 2_BuildDatabases: K} Check Biml For Errors 
D 3.BuildSchemas.b 

O 4 BuildTablesAnd © Open 





Figure 10-44, Generating SSIS packages 


As before, when generating the Destination Tables SSIS package, you will 
be prompted to overwrite the AdventureWorks2014 Stage project connection 
manager and its file. The process generates 75 SSIS packages in all: 


1 Create Destination database 
1 Create Destination schemas 
1 Create Destination tables 


71 SSIS loaders, one for each table in the 
AdventureWorks2014 database 


1 controller that contains 71 Execute Package tasks, 
one for each SSIS loader 
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Please see Figure 10-45. 






ceG|o-s@als— 


Search Solution Explorer (Ctri+;) p- 






Aw BimlkExpressMetadataFramework (SQL Server 2014) 
@ Project.params 
4 tm) Connection Managers 
# _master_AdventureWorks2014_Stage.conmgr 
¥ AdventureWorks2014_Source.conmgr 
@ AdventureWorks2014_Stage.conmgr 
4 ta) SSIS Packages 
E 10 Build_Destination_Databases.dtsx 
A 20 Build Destination Schemas.dtsx 
A 30_Build_Destination_Tables.dtsx 
E, 99 Execute AdventureWorks2014_Stage_ Loader Staging_Packages.dtsx 
EL AdventureWorks2014_Staging_dbo_AWBuildVersion_TruncateAndLoad.dtsx 
E AdventureWorks2014_Staging_dbo_DatabaseLog_TruncateAndLoad.dtsx 
E AdventureWorks2014_Staging_dbo_ErrorLog_TruncateAndLoad.dtsx 
A AdventureWorks2014_Staging_HumanResources_Department_TruncateAndLoad. 
E AdventureWorks2014_Staging_HumanResources_Employee_TruncateAndLoad.dt 
E, AdventureWorks2014_Staging_HumanResources_EmployeeDepartmentHistory_Ti 
E} AdventureWorks2014_Staging_HumanResources_EmployeePayHistory_Truncateé 
E AdventureWorks2014_Staging_HumanResources_JobCandidate_TruncateAndLoa: 
E, AdventureWorks2014_Staging_HumanResources_Shift_TruncateAndLoad.dtsx 
E, AdventureWorks2014_Staging_Person_Address_TruncateAndLoad.dtsx 
E) AdventureWorks2014_Staging_Person_AddressType_TruncateAndLoad.dtsx 
E, AdventureWorks2014_Staging_Person_BusinessEntity_TruncateAndLoad.dtsx 
E, AdventureWorks2014_Staging_Person_BusinessEntityAddress_TruncateAndLoad.c 
E, AdventureWorks2014_Staging_Person_BusinessEntityContact_TruncateAndLoad.¢ 
E) AdventureWorks2014_Staging_Person_ContactType_TruncateAndLoad.dtsx 
E, AdventureWorks2014_Staging_Person_CountryRegion_TruncateAndLoad.dtsx 
E AdventureWorks2014_Staging_Person_EmailAddress_TruncateAndLoad.dtsx 
E, AdventureWorks2014_Staging_Person_Password_TruncateAndLoad.dtsx 
A AdventureWorks2014_Staging_Person_Person_TruncateAndLoad.dtsx 
E, AdventureWorks2014_Staging_Person_PersonPhone_TruncateAndLoad.dtsx 
E} AdventureWorks2014_Staging_Person_PhoneNumberlype_TruncateAndLoad.dts: 
EL AdventureWorks2014_Staging_Person_StateProvince_TruncateAndLoad.dtsx 





Figure 10-45. AdventureWorks2014 Staging SSIS complete! 
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Executing the Controller 


Execute the Controller SSIS package named 99 Execute AdventureWorks 
2014 Stage Loader Staging Packages.dtsx, as shown in Figure 10-46. 


| BimlExpressMetadataFramework (Running) - Microsoft Visual Studio 

File Edit View Project Build Debug Team Format 555 Tools Test BimlExpress Analyze Wint 
£@-0O/ 8 ~ So Hd| - Q ~| Develop ~ Default Continue -| SG Lf um 6|‘ 
Š Process: [7272] DtsDebugHost.exe = 0 Lifecycle Events = Thread: - 







x | AdventureWorks201...Load.dtsx [Design] AdventureWorks201...Load.dtsx 


gy DataFlow W Parameters [F] EventHandiers “i= Package Explorer (È Progress 


Ww 


A Execute AdventureWorks2014_Staging_Person_Address_TruncateAndLoad Package 


Q 


» ; 
Fa Execute AdventureWorks2014_Staging Person_AddressType_TruncateAndLoad Package 
E 


Q 


(Bal Execute AdventureWorks2014_Staging_dbo_AWBuildVersion_TruncateAndLoad Package 


w 


(il Execute Adventuretvorks2014_5taging_Production_BillofMaterials_TruncateAndLoad Package 


w 


3 
A Execute AdventureWorks2014_5taging_Person_BusinessEntity_TruncateAndLoad Package 


v” 


Connection Manages 
E (project)__master_AdventureWorks2014_Stage  ¥ (project) AdventureWorks2014_Source  ¥ (project) AdventureWorks2014_Stage 


Call Stack Breakpoints Exception Settings Command Window Immediate Window Output ErrorList Autos Locals Wa 


Read 





Figure 10-46. Executing the Controller SSIS package 
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Validating the Load 


In SSMS, run a few test queries to validate the load process executed 
successfully, as shown in Figure 10-47. 


1 Use AdventureWorks2014 Staging 
2 go 
3 Select Count(*) From Person.Person 


100 % we 
EB Results Gil Messages 


(No column name) 


1 | 19972 


Rececceee ees ssessseeeeeesseeseeeeeesssssssnseeeeeee 





Figure 10-47. Validating the load 


The load process is validated. 
You should now be able to deploy the ISPAC file for the 
BimlExpressMetadataFramework SSIS project, as shown in Figure 10-48. 


ware View 


BimlExpressMetadataFramework > bin > Development 


A 


Name 


2 BimlExpressMetadataFramework.ispac 


Open 
7-Zip 
CRC SHA 


Figure 10-48. Preparing to deploy the BimlExpressMetadata 
Framework ISPAC file 
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Full Circle 


You've come full circle in this demo. You deployed an SSIS project to an 
SSIS catalog in Chapter 3. In Chapter 7, you executed a script that added 
metadata to SSIS Framework Community Edition for an SSIS catalog 
project, an SSIS project already deployed to an SSIS catalog, which 
allowed you to execute all of the SSIS packages that were part of the SSIS 
catalog project by calling a single stored procedure and passing it a single 


parameter value. 
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Conclusion 


As shipped, SQL Server Integration Services (SSIS) is ready for enterprise 
data integration, but not as ready for enterprise DevOps and data 
integration lifecycle management (DILM). Using the best practices, 
tools, and utilities outlined in this book, you can deliver data integration 
solutions that participate in enterprise DevOps. 

The collection of free, free and open-source, and for-sale utilities 
available at DILM Suite are built to support enterprise data integration 


lifecycle management for enterprises using SSIS for data integration. 
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APPENDIX A 


Links 


SQL Server Developer Edition 
microsoft.com/en-us/sql-server/application-development 
SQL Server Management Studio (SSMS) 


docs.microsoft.com/en-us/sql/ssms/download-sql- 
Server-management-studio-ssms 


SQL Server Data Tools (SSDT) 


docs.microsoft.com/en-us/sgql/ssdt/download-sql-server-data- 


tools-ssdt 


SQL Server Central Stairway to Integration Services 
sqlservercentral.com/stairway/72494/ 


Visual Studio Online 

visualstudio.com 

Team Foundation Services (TFS) 
visualstudio.com/en-us/docs/tools 

“Deploying SSIS Projects to a Restored SSIS Catalog (SSISDB)” 


andy leonard. blog/2017/07/29/deploying-ssis-projects-to- 
a-restored-ssis-catalog-ssisdb 


© Andy Leonard 2018 
A. Leonard, Data Integration Life Cycle Management with SSIS, 
https://doi.org/10.1007/978-1-4842-3276-7_12 
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Catalog Browser 


dilmsuite.com/catalog-browser 


SSIS Catalog Compare 
dilmsuite.com/ssis-catalog-compare 

SSIS Framework Community Edition 
dilmsuite.com/ssis-framework-community-edition 


“Adding an SSIS Application to SSIS Framework Community Edition’ 


andy leonard. blog/2017/07/26/adding-an-ssis-application- 
to-ssis-framework-community-edition 


Framework Browser 
dilmsuite.com/framework-browser 
Catalog Reports 
dilmsuite.com/catalog-reports 

Biml 
varigence.com/biml and bimlscript.com 
BimlExpress 

varigence.com/bimlexpress 
BimlExpress Metadata Framework 
dilmsuite.com/biml-express-metadata- framework 
AdventureWorks2014 


msftdbprodsamples.codeplex.com/downloads/get/880661 
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73-74 

configuring reference, 74-76 
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reference mapping, 71, 76-77 

SSIS catalog environment, 
69-70 

overriding 

catalog project literal 
override, 67 

ConnectionString property 
value, 66 

ConnectionString’s Value, 65 

design-time default, 68-69 

value text decoration, 68 


D 
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48-49 


SSIS project deployed, 48 
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DILMSample SSIS project, 8 
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execute SQL task, 12-15 
lifecycle management, 4 
packages, 123 
script task 
C#, 16-17 
.Net Code Compiles, 
testing, 20-22 
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